- Email: [email protected]
Human factors approach to evaluate the user interface of physiologic monitoring
Human Factors Approach to Evaluate the User Interface of Physiologic Monitoring Richard Fidler, Raymond Bond, Dewar Finlay, Daniel Guldenring, Anthony Gallagher, Michele Pelter, Barbara Drew, Xiao Hu PII: DOI: Reference:
S0022-0736(15)00289-7 doi: 10.1016/j.jelectrocard.2015.08.032 YJELC 52129
To appear in:
Journal of Electrocardiology
Please cite this article as: Fidler Richard, Bond Raymond, Finlay Dewar, Guldenring Daniel, Gallagher Anthony, Pelter Michele, Drew Barbara, Hu Xiao, Human Factors Approach to Evaluate the User Interface of Physiologic Monitoring, Journal of Electrocardiology (2015), doi: 10.1016/j.jelectrocard.2015.08.032
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT 1 Human Factors Approach to Evaluate the User Interface of Physiologic Monitoring
RI P
Gallagher3, Michele Pelter1, Barbara Drew1, Xiao Hu1
T
Richard Fidler1,2; Raymond Bond3, Dewar Finlay3, Daniel Guldenring3, Anthony
SC
1. University of California San Francisco, School of Nursing, Department of
NU
Physiological Nursing, San Francisco, CA, USA
2. San Francisco VA Medical Center, San Francisco, CA, USA
MA
3. University of Ulster, Ulster, Ireland, UK
CE
PT
ED
Corresponding Author: Richard Fidler, PhD, MBA, CRNA 4150 Clement Street San Francisco VA Medical Center, Anesthesiology (129) San Francisco, CA 94121, USA Tel: 415-750-2069 Fax: 415-750-6653 Email: [email protected]
AC
Running Title: Human Factors Monitoring User Interface Evaluation
ACCEPTED MANUSCRIPT 2 Abstract
AC
CE
PT
ED
MA
NU
SC
RI P
T
Background As technology infiltrates more of our personal and professional lives, user expectations for intuitive design have driven many consumer products, while medical equipment continues to have high training requirements. Not much is known about the usability and user experience associated with hospital monitoring equipment. This pilot project aimed to better understand and describe the user interface interaction and user experience with physiologic monitoring technology. Design This was a prospective, descriptive, mixed-methods quality improvement project to analyze perceptions and task analyses of physiologic monitors. Methods Following a survey of practice patterns and perceived abilities to accomplish key tasks, 10 voluntary experienced physician and nurse subjects were asked to perform a series of tasks in 7 domains of monitor operations on GE Monitoring equipment in a single institution. For each task analysis, data were collected on time to complete the task, the number of button pushes or clicks required to accomplish the task, economy of motion, and observed errors. Results Although 60% of the participants reported incorporating monitoring data into patient care, 80% of participants preferred to receive monitoring data at the point of care (bedside). Average perceived central station usability is 5.3 out of 10 (ten is easiest). Conclusions High variability exists in monitoring station interaction performance among those participating in this project. Alarms were almost universally silenced without cognitive recognition of the alarm state. Education related to monitoring operations appeared largely absent in this sample. Most users perceived the interface to not be intuitive, complaining of multiple layers and steps for data retrieval. These clinicians report real-time monitoring helpful for abrupt changes in condition like arrhythmias; however, reviewing alarms is not prioritized as valuable due to frequent false alarms. Participants requested exporting monitoring data to electronic medical records. Much research is needed to develop best practices for display of real-time information, organization and filtering of meaningful data, and simplified ways to find information. Keywords central station; bedside monitoring, human factors in monitoring; usability; user interface; user experience
ACCEPTED MANUSCRIPT 3
Introduction
T
As technology has proliferated in modern society, consumers have
RI P
developed an impressive level of proficiency for using technology. Ubiquitous technology in smart phones, personal computers, and consumer electronics has
SC
changed the user’s expectations for performance and integration of technology in
NU
our daily lives at work and home1. Intuitive design and usability has become an expected feature of consumer electronics, particularly in the interaction design of
MA
such technologies2. Although the Food and Drug Administration has guidance for the application of human factors and usability engineering to optimize medical
ED
device design, the methods of testing, evaluation, and benchmarking are loosely
PT
delineated3.
This project was conceived and designed to explore the current state of
CE
real-world usability of bedside and central monitoring in a hospital. The aims of
AC
this work were (1) to gain insight into whether usability issues exist in hospital monitoring equipment, (2) gather opinions and perceptions about physiological monitoring, (3) explore barriers and successes of current equipment in multidisciplinary practice, and (4) to report this unbiased information to the scientific community to improve the user interface design and functionality of physiological displays. Methods Design This was a prospective, descriptive, observational human factors approach to
ACCEPTED MANUSCRIPT 4 understanding physiologic monitoring usability. This included task analyses, timemotion analyses, and user perception surveys.
T
Equipment
RI P
The equipment used by participants in this study included General Electric (GE) Solar 8000i at the bedside in the intensive care, GE Dash 4000 in the
SC
emergency and telemetry units, and GE Central Monitoring Stations. All of the
NU
bedside monitors are connected to the central monitoring station in that clinical area. This particular equipment was installed in the intensive care unit in 2007,
MA
the telemetry step-down unit in 2010, and the emergency department in 2011. Data Collection
ED
Tasks for analyses were compiled by a group of ECG researchers and
PT
experts from the US, UK and Ireland. Each task was considered a basic user function for clinical physiological monitoring, and actual patients being monitored
CE
were used. For each task analysis, the participant was asked to rate the
AC
perceived difficulty of the task on the 0 to 10 Likert scale 4, with 10 being the easiest and 0 being so difficult that they do not think that they can perform the task. Next, the participant was instructed to select a currently monitored patient, perform the task, and then reset the patient monitoring settings back to the pretest settings. Times were measured for task performance by 2 raters using digital stopwatches. Also, the number of button pushes, screen touches, and mouse clicks were recorded. Monitoring tasks that were included in this project represented seven domains in monitoring functionality: 1. Alarms silencing.
ACCEPTED MANUSCRIPT 5 2. Alarms and waveform review. 3. Trends display for vital signs.
T
4. Parameter alarm adjustments.
RI P
5. Pacing detection settings. 6. ST-segment monitoring practices.
MA
NU
Task Analysis Data Collection Methods
SC
7. Respiratory rate monitoring (impedance) settings.
1. Alarm Silencing
ED
Participants were asked to silence an active monitoring alarm. The alarms
PT
may be silenced by either the keyboard button or via the monitor touchscreen, so this was recorded as part of the user interaction. As part of the task analysis,
CE
after the alarm was silenced, each participant was asked to turn away from the
AC
monitor. Without looking back, the participant was asked to report which patient or what type of alarm was silenced, or both 2. Alarms and Waveform Review Participants were asked to select a patient and display all of the arrhythmia alarms for the past 24 hours with metrics recorded. After closing this patient screen, the participants were presented a scenario where the patient may have changed rhythm in the past hour, and the task was to display the ECG waveform from approximately one-hour ago. The third task in this domain asked the participant to display and print a multi-lead ECG waveform for any of the
ACCEPTED MANUSCRIPT 6 arrhythmia alarms for the selected patient. 3. Trends Display for Vital Signs
T
Participants were asked to select a patient and display a 24-hour heart
RI P
rate trend. The heart rate trend could be from either the ECG or pulse oximetry signals. After the 24-hour heart rate trend was displayed, the participant was
SC
asked to display a 24-hour trend for either blood pressure or pulse oximetry to
NU
determine whether intra-task learning was conferred. 4. Parameter Alarms Adjustment
MA
Participants were given the scenario of a patient with sinus tachycardia triggering the high heart rate alarm, and they were tasked with adjusting the high
ED
heart rate limit upward by 10 beats per minute. Subjects were then asked to
PT
adjust the pulse oximetry lower limit to 88%. Since atrial fibrillation often triggers heart rate parameter alarms, adjusting the atrial fibrillation or irregular heart rate
AC
section.
CE
setting from an audible alarm to message notification was included in this
5. Pacing Detection Participants were asked to select a patient that they believe to have either a temporary or permanent pacemaker. They were then asked to activate the pacing detection feature for that patient. Subjects were asked to explain what activating the pacing detection feature does to the displayed ECG waveform, and they were asked to describe the differences between the Pace 1 and Pace 2 settings. 6. ST-Segment Monitoring Practices
ACCEPTED MANUSCRIPT 7 Subjects were asked to perform three distinct tasks related to ST-segment monitoring on three different patients. The first task was to change the ST-
T
elevation limit to 3 mm in a single lead. The second task was to change the ST-
RI P
depression limit to -3 mm in a single lead. The third task was to adjust STsegment parameters consistent with electrocardiographic monitoring standards in
SC
hospitals5. Subjects were asked to adjust the ST-elevation and depression
NU
settings 1 mm above and below a selected patient's ST-segment baseline. In clinical practice, the ST-segment settings should be tailored in all of the
MA
monitoring leads; however, due to time constraints for participants in this study, only the tailored adjustment of a ECG lead V1 was performed for analysis.
ED
7. Respiratory Rate Monitoring (by impedance) Settings
PT
Subjects were asked to select a patient, then change the respiratory lead
CE
from the factory default of ECG lead II to lead I.
AC
As a mixed method approach, both qualitative and quantitative data were collected to provide greater insight into man-machine interactions. Qualitative data was gathered in individual interview style recorded by the observing researchers. Qualitative data included the participant's use of monitoring in practice, whether prior training has been received, and perceptions regarding the difficulty or ease of each of the tasks being quantitatively analyzed. Quantitative data included information about perceived difficulty on a Likert scale4, where zero represents a task that is difficult, insomuch that the user knows that they cannot perform the task, and the maximum of the scale is 10, where the user feels
ACCEPTED MANUSCRIPT 8 completely confident in being able to perform the task quickly, likely because they perform this task on a regular basis.
T
Quantitative data was gathered through observation. This included task
RI P
completion times and economy of motion measured by the number of screen touches, button pushes, or mouse clicks required for task completion. To improve
SC
accuracy of observations, lead author RF was the primary observer and
NU
interviewer with a master's prepared registered nurse serving as a second observer and recorder for real-time data collection and entry. A pair of electronic
MA
stopwatches were used, and the average of the two measured times were used for data analysis.
ED
This project focused on the central monitoring station, although some
PT
tasks were also conducted on the bedside monitors in the patient rooms. Data were also collected on interaction performance while patient monitoring was
CE
occurring in real-time. Thus, all monitoring adjustments that were part of the task
AC
analyses on 138 patients were reset to the original state prior to moving to the next task. Participants were instructed that they could choose to defer any task for any reason.
Data were collected in April 2015 at a single teaching hospital in San Francisco, CA over a 2-week time interval. Subjects were randomly selected volunteers that were informed that data was being collected about the use of physiological monitors and their usability. All of the participants were experienced clinicians in critical care, emergency care, and telemetry progressive care types of practice. For the purposes of this project, a set of tasks was constructed by
ACCEPTED MANUSCRIPT 9 consensus involving multiple experts and researchers with expertise in physiological monitoring.
T
Results
RI P
Sample Characteristics
This experiment involved the recruitment of 10 volunteer subjects. All
SC
participants were experienced in their respective profession with 9.2 mean years
NU
of experience (range 4-16 years, SD 4.4). Each participant was given the same surveys, and each participated fully in the data collection methodologies.
MA
Participants took an average of 24 minutes to complete the tasks and questions; however, there was wide variation in the time to complete with some taking as
PT
ED
little as 20 minutes but as much as 46 minutes.
CE
Perceptions of Usability
AC
For the central monitoring station, participants reported that they interact with the monitoring equipment a mean of 3.2 (range 0-10) times per day. In this sample, only 6 participants reported that they routinely incorporate physiologic monitoring data into their daily patient assessments. Barriers to incorporation of monitoring data into daily patient rounds and management included comments regarding time to retrieve data, the high number of false alarms, lack of familiarity with the types of data and the methods for data retrieval from the monitoring central station, and all participants remarked that inclusion of monitoring data into the electronic medical record (EMR) would simplify migration of monitoring data
ACCEPTED MANUSCRIPT 10 into daily assessments. The monitoring data requested to include in the EMR includes vital signs trends, arrhythmia waveforms, and alarms information.
T
Participants stated that they are required to interact with the EMR multiple times
RI P
per day for patient record entry, so they prefer the EMR venue for data review in a single location.
SC
The Likert scale perceived usability of the central monitoring station was
NU
reported as moderately difficult, with a mean of 5.3 (SD 1.7) and a mode of 5 on the 10-point Likert scale. Only one participant had onsite training provided by the
MA
manufacturer when the equipment was installed several years prior. Two subjects stated that they had on-the-job training on using the monitoring
ED
equipment, but also reported that the training lasted 10 minutes and 15 minutes
PT
respectively. The remaining seven subjects had no training on the monitoring equipment. As part of the task analysis, it was noticed that none of the 138
CE
unique patients randomly selected for monitor setting manipulation had their
AC
settings adjusted from the manufacturer's factory default settings.
Subjective comments from participants were collected, including the response that all participants prefer to obtain information from the monitor at the patient bedside. Other comments were that a picture or video of the patient on the central station next to their physiological data would improve the clinicians' recall of that patient. A video would also help to determine the visual characteristics of a patient (e.g. whether they are moving, resting or in distress). Task Analysis Results
ACCEPTED MANUSCRIPT 11
1. Alarms Silencing
T
The alarm was silenced by use of keyboard by two subjects, and the other
RI P
8 subjects used the touchscreen. The mean time to silence the alarm measured from the time of the instruction to the absence of the audible alarm was 6.2
SC
seconds (SD 2.3). The perceived difficulty of this task was a mean of 8.9 (SD
NU
2.3) with a mode response rating of 10, implying that users believe that this is a very simple task. Of the 10 participants, only one knew the alarm type and
MA
patient whose alarm was silenced. No other participant could state the alarm type or patient with the alarm.
ED
2. Alarms and Waveform Review
PT
The perceived difficulty on the Likert scale was a mean of 8.0 (SD 3.2) and a mode of 10, rating user perception of performing these tasks as extremely
CE
simple. There was wide variability in task completion times for each task, and the
AC
task of displaying the ECG waveform from one-hour prior was particularly challenging for four subjects. There were subjective comments from users surrounding the themes of the timeline on the screen and that they are usable by touchscreen and thus requires the use of the mouse. Also, users commented that there is no intuitive method for seeing more than the current streaming ECG waveform. Participants also commented that clicking on the ECG waveform on the screen might be an intuitive method for obtaining more ECG waveform data. Comparison of the perceived simplicity of performing this function with the wide variation in task completion times and the number of button pushes suggests a
ACCEPTED MANUSCRIPT 12 lack of user insight into their ability to interact with the monitor interface. 3. Trends Display for Vital Signs
T
The perceived difficulty of displaying a 24-hour trend of heart rate was
RI P
rated as a mean of 7.8 (SD 2.9) with a response mode of 10. This task was achieved with a mean time of 28.6 seconds (SD 19.7) with 6.7 button pushes
SC
(SD 2.3). Participants were subsequently asked to display the trend of either the
NU
blood pressure (7 subjects) or pulse oximetry data (3 subjects) for another patient, taking a mean of 27.1 seconds (SD 30.3) with 7.8 button pushes (SD
MA
8.8). Performing these two tasks is very similar on the user interface evaluated, but the lack of difference in mean time between the first and second trend display
ED
suggests that no immediate intra-task learning occurred (p=0.91). Subjective
PT
comments about this task were centered on the lack of intuitiveness. Participants commented that clicking on the numerical heart rate, blood pressure, or pulse
CE
oximetry value should result in the display of the trend.
AC
4. Parameter Alarm Adjustment The perceived confidence of being able to perform this task was variable with a mean Likert scale score of 6.4 (SD 4.5) out of 10, and response mode of 10. Subjective comments provided by participants centered on automated adjustment of parameter alarms. The clinicians providing comments suggested that the monitor should automatically create a parameter range around the patient's vital signs and only alert clinicians of changes in the individualized measures for each patient. 5. Pacing Detection Settings
ACCEPTED MANUSCRIPT 13 The perceived confidence of being able to perform this task was variable with a mean Likert scale response of 4.7 (SD 5.1) with a mode of 10, and
T
performance took a mean of 18.4 seconds (SD 17.0) with 2.9 button pushes (SD
RI P
1.6). While conducting this task, participants were asked to distinguish whether they would activate the "Pace 1" or "Pace 2" settings that are offered in the user
SC
interface. There were two participants that knew that there is a difference
NU
between these two options, but neither of them could remember what the difference is. Subjects recommended manufacturers label them as to their actual
MA
function instead of Pace 1 and Pace 2. When asked what activation of pacing detection does for the monitoring system, only one participant provided a
ED
response stating, "The monitor puts a white spike on the ECG tracing".
PT
6. ST-segment Monitoring Practices All of the monitoring systems in this study are capable of continuous ST-
CE
segment monitoring. In this data, only the intensive care unit had activated any
AC
ST-segment monitoring on certain patients, while none of the monitors in the progressive care or emergency departments had any ST-segment monitoring activated. The perceived difficulty of these tasks was rated as moderately challenging on the zero to 10 Likert scale with a mean perceived difficulty of 5.9 (SD 4.6) and a mode of 10 by 2 subjects. Three of the 10 subjects did not complete all of the tasks due to reported difficulty and frustration. Adjusting the upper limit of ST-segment to 3mm above baseline, taking a mean of 25.8 seconds (SD 15.7) with 9.1 button pushes (SD 3.3). Subsequent adjustment of the ST-depression alarm to 3mm below the baseline in ECG lead
ACCEPTED MANUSCRIPT 14 III took a mean time of 27.0 seconds (SD 38.5) with 7.3 button pushes (SD 4.5). Consistent with current ECG monitoring guidelines, the participants were tasked
T
with adjusting the ST-segment alarms to 1mm above and 1mm below the
RI P
baseline in ECG lead V1, which took a mean time of 116.7 seconds (SD 18.6) and 20.6 button pushes (SD 6.5). The subjects relate part of the increase in time
SC
to completion of the final task in this domain to the identification of the ST-
NU
segment baseline. This represents the adjustment of only one monitoring lead to practice guidelines, and this may represent one of the challenges to instituting
MA
ST-segment monitoring.
Subjective responses by the three participants who did not complete the
ED
task included remarks about ST-segment monitoring being time consuming,
PT
creating too many monitoring alarms, and too time consuming to perform the settings. Of the participants that completed the task, all of them remarked that
CE
they felt that the monitor should be able to do this automatically, and that
AC
completing this task for relevant monitoring leads would just be too time consuming for all of the patients. 7. Respiratory Rate Monitoring by Impedance Participants were asked to use the central monitoring station to evaluate the respiratory waveform and to change the ECG monitoring lead for respiratory rate from lead II to lead I. Unknown to this team, changing leads for respiratory rate monitoring could only be done on bedside equipment. All participants were given the opportunity to perform the task on the bedside monitor, which all remarked that it was much easier to do this at the bedside. When asked about
ACCEPTED MANUSCRIPT 15 the use of impedance technology for respiratory rate monitoring, none of the participants were aware of how the monitor measured respiratory rate. Eight of
RI P
movement of the electrodes to measure respiratory rate”.
T
the ten participants stated that they believed that the monitor “detected
Discussion
SC
Consumer electronics have become popular due to their user-friendliness
NU
and intuitive designs. However, these characteristics are a result of large investments made to research and development in the area of human factors,
MA
human computer interaction and to the User Experience (UX) of the interface. In fact many electronic devices such as smart phones have a myriad of advanced
ED
features and yet general members of the public can adopt these technologies
PT
with ease and without the use of large instruction manuals2. This is mainly due to the implementation of UX principles such as affordance, i.e. the device should
CE
behave as expected or anticipated. For example, if the majority of users expect
AC
to see more electrocardiographic options when they click on the ECG signal – then this should be the case. Whilst popular electronic devices are dedicated to usability engineering, perhaps manufacturers of medical devices are a little behind. This has been recognized by regulatory bodies such as the FDA who now enforce medical devices to undergo a ‘usability validation’ before they are introduced into real clinical scenarios3,5. This is of great importance given suboptimal usability of medical devices can result in intragenic errors and a threat to patient safety. Limitations
ACCEPTED MANUSCRIPT 16 The main limitation of this experiment is the small sample size. However, according to an empirical function composed by usability experts, 80% of
T
usability issues can be uncovered by recruiting just five subjects6. Nevertheless,
RI P
we recognize that we would need more participants to gain a better representation of other quantitative data such as task completion times. Also, this
SC
study was conducted at a single teaching hospital using only one brand of
NU
monitoring equipment-limiting generalizability. These authors also recognize that 70% of the participants had no formal training in the use of these monitors, and
MA
although that is acceptable at this facility, it is not clear what training practices exist regarding monitoring equipment at other facilities warranting further
ED
research.
PT
Conclusions
Our impetus for embarking in this study was to gain insight into whether
CE
usability issues exist in hospital monitoring, and we conclude that physiological
AC
monitors should be improved regarding their clinician-friendliness and interaction design. This is evidenced by the fact that participants failed to complete a number of tasks and/or completed tasks within an extended amount of time. In addition, there was also a lack of confidence and competence in using a number of features of the monitor. This included a lack of proficiency in setting personalized thresholds for alarms. This illustrates that clinicians do not exploit the capability of medical machines. This fact should be considered when developing new functionalities for physiological monitors and more effort should be made to increase the uptake of current monitoring features.
ACCEPTED MANUSCRIPT 17 We also aimed to gather opinions and perceptions about physiologic monitors, and we found that monitor credibility is compromised by high false
T
alarm rates, and clinicians would prefer a single point of access for all patient
RI P
data--specifically full monitoring data available through the electronic health record. Multiple subjects cited this point as a barrier to incorporating monitoring
SC
data into clinical rounds and patient management. We recommend that hospitals
NU
enhance the competency of nurses in using physiological monitors through simulation-based training that would include scenarios where features such as
MA
ST-segment monitoring are of critical importance. Future work will include the recruitment of additional participants to validate task completion times. We will
ED
also consider video recording tasks for annotation purposes and for analyzing
James, J. 10 ways developers can meet user expectations and ease
AC
1.
CE
References
PT
specific events and usability issues.
frustrations. TechRepublic/US. 13 May 2009. Web. 7 July 2015. 2.
Monsees, J. Putting "intuitive" back into intuitive design. UX Magazine. 14
October 2014. Web 7 July 2015. 3. United States. Food and Drug Administration. Draft Guidance for Industry and Food and Drug Administration Staff-Applying Human Factors and Usability Engineering to Optimize Medical Device Design. Food and Drug Administration, 2011. http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDoc
ACCEPTED MANUSCRIPT 18 uments/ucm259748.htm 4. Likert, R. (1932). A Technique for the Measurement of Attitudes. Archives of
T
Psychology, 140, 1–55.
RI P
5. Drew, B., Califf, R., Funk, M., Kaufman, E., Krucoff, M., Laks, M., Macfarlane, P., Sommargren, C., Swiryn, S. & Van Hare, G. (2004). Practice Standards for
SC
Electrocardiographic Monitoring in Hospital Settings. Circulation; 110: 2721-
6.
NU
2746. doi: 10.1161/01.CIR.0000145144.56673.59.
FDA, Human Factors Implications of the New GMP Rule Overall
MA
Requirements of the New Quality System Regulation, 2010. Available: http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/PostmarketR
Nielsen, J. and Landauer, T.K. A mathematical model of the finding of
PT
7.
ED
equirements/Human Factors/ucm119215.htm (01.11.10).
usability problems. In Proceedings of INTERCHI 1993 (Amsterdam, the
AC
CE
Netherlands, Apr. 24–29). ACM Press, New York, 1993, 206–213.
ACCEPTED MANUSCRIPT 19 Table 1. Test Domains and Tasks Performed
SC
RI P
2. Alarms and waveform review.
MA
ED PT
AC
CE
5. Pacing Detection
NU
3. Trends Display for Vital Signs
4. Parameter Alarms Adjustment
6. ST-Segment Monitoring Practices
7. Respiratory Rate Monitoring by Impedance
1. Identify a patient with a monitoring alarm then silence the alarm (keyboard or touchscreen). 2. Ask participant to identify the patient or the alarm type or both. 1. Select a patient and display all of the arrhythmia alarms for the past 24-hours. 2. Display the ECG waveform from 1-hour ago. 3. Print a hard copy of multi-lead ECG for an ECG arrhythmia alarm condition. 1. Select a patient and display a 24-hour trend for heart rate (both ECG or pulse oximetry acceptable). 2. Display a 24-hour trend for either blood pressure or pulse oximetry. 1. Select a patient and increase the high heart rate alarm limit by 10 beats per minute. 2. Select another patient and decrease the low pulse oximetry alarm limit to 88%. 3. Select another patient and change the atrial fibrillation alarm from audible to an inaudible visual message. 1. Identify and select a patient that has a permanent or temporary pacemaker. 2. Activate the pacing detection feature for this patient (or verify it is active if already activated). 3. Ask participant to explain the difference between Pace 1 and Pace 2 functions. 1. Select a patient and change the STsegment elevation limit to 3mm in a single ECG lead. 2. Change the ST-depression limit to 3mm in a single ECG lead. 3. Consistent with current guidelines, adjust the ST-segment alarm thresholds to 1mm above and 1mm below the patient baseline in ECG lead V1. 1. Select a patient and adjust the respiratory rate detection lead from lead II to lead I.
T
1. Alarm Silencing
ACCEPTED MANUSCRIPT
Profession Physician
2
46
Male
Physician
3
47
Male
Physician
4 5 6 7 8 9 10
28 30 35 63 34 29 41 Mean 39.4
Female Female Male Female Female Female Female 70% Female
Nurse Nurse Nurse Nurse Nurse Nurse Nurse 70% Nurse
Range 28-63
AC
CE
PT
RI P
SC
NU
ED
SD 11.6
T
Table. 2 Sample Characteristics Subject Age (years) Gender 1 37 Male
MA
20
Specialty Pulmonary Critical Care Anesthesia Critical Care Pulmonary Critical Care Telemetry Telemetry Telemetry Telemetry Emergency Critical Care Critical Care 3 Critical Care Physicians 4 Telemetry Nurses 3 Emergency or Critical Care Nurses
ACCEPTED MANUSCRIPT 21 Table. 3 Clinician Perceptions of Monitoring Usability
Mean (SD) Mode
1. Alarm Silencing 2. Alarms and Waveform Review 3. Trends Display for Vital Signs 4. Parameter Alarm Adjustment 5. Pacing Detection Settings 6. ST-Segment Monitoring Practices 7. Respiratory Rate Monitoring by Impedance 8. Overall perceived usability of central monitoring station.
8.9 (2.3) 8.0 (3.2) 7.8 (2.9) 6.4 (4.5) 4.7 (5.1) 5.9 (4.6) Insufficient data 5.3 (1.7)
AC
CE
PT
ED
MA
NU
SC
RI P
T
Perceived Task Difficulty
10 10 10 10 10 10 5
ACCEPTED MANUSCRIPT 22 Table 4. Task Analysis Data Summary.
Review ECG Alarms for past 24h
Display ECG waveform from 1hr ago
PT
Display HR Trend past 24h
CE
3. Trends Display for Vital Signs
AC
Display 24h Trend of Blood Pressure or Pulse Oximetry
4. Parameter Alarms Adjustment
Increase high heart rate alarm by 10 beats/min
Decrease low pulse oximetry alarm to 88%
Change atrial fibrillation to messaging
5. Pacing
(1-17.2) 2.3 20.3
1 0 5.9
(8.1-47.6) 14.7 81.4
(14-150.9) 52.3 34.4
Activate pacing
T
1
ED
Print multi-lead ECG hardcopy for an arrhythmia alarm
6.2
Comments
RI P
2. Alarms and Waveform Review
Range SD
Mean Number of Button Pushes Range SD
SC
Silence an alarm
Mean Time (seconds)
NU
1. Alarm Silencing
Task Performed
MA
Test Domain
(2-17) 4.6 15
(5-36) 9.3 8.2
(12.6-75.5) 20.0 28.6
(3-25) 6.6 6.7
(12.7-75.4) 19.7 27.1
(4-12) 2.3 7.8
(4-94.9) 30.3 15.3
(2-26) 8.8 6.3
(6.1-24.3) 5.4 28.1
(4-8) 1.2 9.3
(10.2-109.8) 33.4 16.9
(3-26) 7.2 6.0
(6-31) 8.6 18.4
(2-9) 2.1 4.9
One subject unable to complete after 140 seconds
ACCEPTED MANUSCRIPT 23 Detection
detection
Change ST Elevation Alarm to 3mm
(3-8) 1.6 9.1
(16.5-62.3) 15.7 27.0
(5-13) 3.3 7.3
Change respiratory rate lead from II to I
AC
CE
PT
7. Respiratory Rate Monitoring
SC
NU
(97.8-138.0) 18.6 Unable to complete from central station
ED
Change STSegment Alarms to Practice Guideline
(7.8-122.6) 38.9 116.7
MA
Change ST Depression Alarm to -3mm
RI P
T
6. ST-Segment Monitoring Practices
(6.1-48.7) 17.0 25.8
(5-18) 4.5 20.6
(12-30) 6.5
One subject unable to complete any of these tasks; two more subjects were unable to complete task 3 in this domain
S0022-0736(15)00289-7 doi: 10.1016/j.jelectrocard.2015.08.032 YJELC 52129
To appear in:
Journal of Electrocardiology
Please cite this article as: Fidler Richard, Bond Raymond, Finlay Dewar, Guldenring Daniel, Gallagher Anthony, Pelter Michele, Drew Barbara, Hu Xiao, Human Factors Approach to Evaluate the User Interface of Physiologic Monitoring, Journal of Electrocardiology (2015), doi: 10.1016/j.jelectrocard.2015.08.032
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT 1 Human Factors Approach to Evaluate the User Interface of Physiologic Monitoring
RI P
Gallagher3, Michele Pelter1, Barbara Drew1, Xiao Hu1
T
Richard Fidler1,2; Raymond Bond3, Dewar Finlay3, Daniel Guldenring3, Anthony
SC
1. University of California San Francisco, School of Nursing, Department of
NU
Physiological Nursing, San Francisco, CA, USA
2. San Francisco VA Medical Center, San Francisco, CA, USA
MA
3. University of Ulster, Ulster, Ireland, UK
CE
PT
ED
Corresponding Author: Richard Fidler, PhD, MBA, CRNA 4150 Clement Street San Francisco VA Medical Center, Anesthesiology (129) San Francisco, CA 94121, USA Tel: 415-750-2069 Fax: 415-750-6653 Email: [email protected]
AC
Running Title: Human Factors Monitoring User Interface Evaluation
ACCEPTED MANUSCRIPT 2 Abstract
AC
CE
PT
ED
MA
NU
SC
RI P
T
Background As technology infiltrates more of our personal and professional lives, user expectations for intuitive design have driven many consumer products, while medical equipment continues to have high training requirements. Not much is known about the usability and user experience associated with hospital monitoring equipment. This pilot project aimed to better understand and describe the user interface interaction and user experience with physiologic monitoring technology. Design This was a prospective, descriptive, mixed-methods quality improvement project to analyze perceptions and task analyses of physiologic monitors. Methods Following a survey of practice patterns and perceived abilities to accomplish key tasks, 10 voluntary experienced physician and nurse subjects were asked to perform a series of tasks in 7 domains of monitor operations on GE Monitoring equipment in a single institution. For each task analysis, data were collected on time to complete the task, the number of button pushes or clicks required to accomplish the task, economy of motion, and observed errors. Results Although 60% of the participants reported incorporating monitoring data into patient care, 80% of participants preferred to receive monitoring data at the point of care (bedside). Average perceived central station usability is 5.3 out of 10 (ten is easiest). Conclusions High variability exists in monitoring station interaction performance among those participating in this project. Alarms were almost universally silenced without cognitive recognition of the alarm state. Education related to monitoring operations appeared largely absent in this sample. Most users perceived the interface to not be intuitive, complaining of multiple layers and steps for data retrieval. These clinicians report real-time monitoring helpful for abrupt changes in condition like arrhythmias; however, reviewing alarms is not prioritized as valuable due to frequent false alarms. Participants requested exporting monitoring data to electronic medical records. Much research is needed to develop best practices for display of real-time information, organization and filtering of meaningful data, and simplified ways to find information. Keywords central station; bedside monitoring, human factors in monitoring; usability; user interface; user experience
ACCEPTED MANUSCRIPT 3
Introduction
T
As technology has proliferated in modern society, consumers have
RI P
developed an impressive level of proficiency for using technology. Ubiquitous technology in smart phones, personal computers, and consumer electronics has
SC
changed the user’s expectations for performance and integration of technology in
NU
our daily lives at work and home1. Intuitive design and usability has become an expected feature of consumer electronics, particularly in the interaction design of
MA
such technologies2. Although the Food and Drug Administration has guidance for the application of human factors and usability engineering to optimize medical
ED
device design, the methods of testing, evaluation, and benchmarking are loosely
PT
delineated3.
This project was conceived and designed to explore the current state of
CE
real-world usability of bedside and central monitoring in a hospital. The aims of
AC
this work were (1) to gain insight into whether usability issues exist in hospital monitoring equipment, (2) gather opinions and perceptions about physiological monitoring, (3) explore barriers and successes of current equipment in multidisciplinary practice, and (4) to report this unbiased information to the scientific community to improve the user interface design and functionality of physiological displays. Methods Design This was a prospective, descriptive, observational human factors approach to
ACCEPTED MANUSCRIPT 4 understanding physiologic monitoring usability. This included task analyses, timemotion analyses, and user perception surveys.
T
Equipment
RI P
The equipment used by participants in this study included General Electric (GE) Solar 8000i at the bedside in the intensive care, GE Dash 4000 in the
SC
emergency and telemetry units, and GE Central Monitoring Stations. All of the
NU
bedside monitors are connected to the central monitoring station in that clinical area. This particular equipment was installed in the intensive care unit in 2007,
MA
the telemetry step-down unit in 2010, and the emergency department in 2011. Data Collection
ED
Tasks for analyses were compiled by a group of ECG researchers and
PT
experts from the US, UK and Ireland. Each task was considered a basic user function for clinical physiological monitoring, and actual patients being monitored
CE
were used. For each task analysis, the participant was asked to rate the
AC
perceived difficulty of the task on the 0 to 10 Likert scale 4, with 10 being the easiest and 0 being so difficult that they do not think that they can perform the task. Next, the participant was instructed to select a currently monitored patient, perform the task, and then reset the patient monitoring settings back to the pretest settings. Times were measured for task performance by 2 raters using digital stopwatches. Also, the number of button pushes, screen touches, and mouse clicks were recorded. Monitoring tasks that were included in this project represented seven domains in monitoring functionality: 1. Alarms silencing.
ACCEPTED MANUSCRIPT 5 2. Alarms and waveform review. 3. Trends display for vital signs.
T
4. Parameter alarm adjustments.
RI P
5. Pacing detection settings. 6. ST-segment monitoring practices.
MA
NU
Task Analysis Data Collection Methods
SC
7. Respiratory rate monitoring (impedance) settings.
1. Alarm Silencing
ED
Participants were asked to silence an active monitoring alarm. The alarms
PT
may be silenced by either the keyboard button or via the monitor touchscreen, so this was recorded as part of the user interaction. As part of the task analysis,
CE
after the alarm was silenced, each participant was asked to turn away from the
AC
monitor. Without looking back, the participant was asked to report which patient or what type of alarm was silenced, or both 2. Alarms and Waveform Review Participants were asked to select a patient and display all of the arrhythmia alarms for the past 24 hours with metrics recorded. After closing this patient screen, the participants were presented a scenario where the patient may have changed rhythm in the past hour, and the task was to display the ECG waveform from approximately one-hour ago. The third task in this domain asked the participant to display and print a multi-lead ECG waveform for any of the
ACCEPTED MANUSCRIPT 6 arrhythmia alarms for the selected patient. 3. Trends Display for Vital Signs
T
Participants were asked to select a patient and display a 24-hour heart
RI P
rate trend. The heart rate trend could be from either the ECG or pulse oximetry signals. After the 24-hour heart rate trend was displayed, the participant was
SC
asked to display a 24-hour trend for either blood pressure or pulse oximetry to
NU
determine whether intra-task learning was conferred. 4. Parameter Alarms Adjustment
MA
Participants were given the scenario of a patient with sinus tachycardia triggering the high heart rate alarm, and they were tasked with adjusting the high
ED
heart rate limit upward by 10 beats per minute. Subjects were then asked to
PT
adjust the pulse oximetry lower limit to 88%. Since atrial fibrillation often triggers heart rate parameter alarms, adjusting the atrial fibrillation or irregular heart rate
AC
section.
CE
setting from an audible alarm to message notification was included in this
5. Pacing Detection Participants were asked to select a patient that they believe to have either a temporary or permanent pacemaker. They were then asked to activate the pacing detection feature for that patient. Subjects were asked to explain what activating the pacing detection feature does to the displayed ECG waveform, and they were asked to describe the differences between the Pace 1 and Pace 2 settings. 6. ST-Segment Monitoring Practices
ACCEPTED MANUSCRIPT 7 Subjects were asked to perform three distinct tasks related to ST-segment monitoring on three different patients. The first task was to change the ST-
T
elevation limit to 3 mm in a single lead. The second task was to change the ST-
RI P
depression limit to -3 mm in a single lead. The third task was to adjust STsegment parameters consistent with electrocardiographic monitoring standards in
SC
hospitals5. Subjects were asked to adjust the ST-elevation and depression
NU
settings 1 mm above and below a selected patient's ST-segment baseline. In clinical practice, the ST-segment settings should be tailored in all of the
MA
monitoring leads; however, due to time constraints for participants in this study, only the tailored adjustment of a ECG lead V1 was performed for analysis.
ED
7. Respiratory Rate Monitoring (by impedance) Settings
PT
Subjects were asked to select a patient, then change the respiratory lead
CE
from the factory default of ECG lead II to lead I.
AC
As a mixed method approach, both qualitative and quantitative data were collected to provide greater insight into man-machine interactions. Qualitative data was gathered in individual interview style recorded by the observing researchers. Qualitative data included the participant's use of monitoring in practice, whether prior training has been received, and perceptions regarding the difficulty or ease of each of the tasks being quantitatively analyzed. Quantitative data included information about perceived difficulty on a Likert scale4, where zero represents a task that is difficult, insomuch that the user knows that they cannot perform the task, and the maximum of the scale is 10, where the user feels
ACCEPTED MANUSCRIPT 8 completely confident in being able to perform the task quickly, likely because they perform this task on a regular basis.
T
Quantitative data was gathered through observation. This included task
RI P
completion times and economy of motion measured by the number of screen touches, button pushes, or mouse clicks required for task completion. To improve
SC
accuracy of observations, lead author RF was the primary observer and
NU
interviewer with a master's prepared registered nurse serving as a second observer and recorder for real-time data collection and entry. A pair of electronic
MA
stopwatches were used, and the average of the two measured times were used for data analysis.
ED
This project focused on the central monitoring station, although some
PT
tasks were also conducted on the bedside monitors in the patient rooms. Data were also collected on interaction performance while patient monitoring was
CE
occurring in real-time. Thus, all monitoring adjustments that were part of the task
AC
analyses on 138 patients were reset to the original state prior to moving to the next task. Participants were instructed that they could choose to defer any task for any reason.
Data were collected in April 2015 at a single teaching hospital in San Francisco, CA over a 2-week time interval. Subjects were randomly selected volunteers that were informed that data was being collected about the use of physiological monitors and their usability. All of the participants were experienced clinicians in critical care, emergency care, and telemetry progressive care types of practice. For the purposes of this project, a set of tasks was constructed by
ACCEPTED MANUSCRIPT 9 consensus involving multiple experts and researchers with expertise in physiological monitoring.
T
Results
RI P
Sample Characteristics
This experiment involved the recruitment of 10 volunteer subjects. All
SC
participants were experienced in their respective profession with 9.2 mean years
NU
of experience (range 4-16 years, SD 4.4). Each participant was given the same surveys, and each participated fully in the data collection methodologies.
MA
Participants took an average of 24 minutes to complete the tasks and questions; however, there was wide variation in the time to complete with some taking as
PT
ED
little as 20 minutes but as much as 46 minutes.
CE
Perceptions of Usability
AC
For the central monitoring station, participants reported that they interact with the monitoring equipment a mean of 3.2 (range 0-10) times per day. In this sample, only 6 participants reported that they routinely incorporate physiologic monitoring data into their daily patient assessments. Barriers to incorporation of monitoring data into daily patient rounds and management included comments regarding time to retrieve data, the high number of false alarms, lack of familiarity with the types of data and the methods for data retrieval from the monitoring central station, and all participants remarked that inclusion of monitoring data into the electronic medical record (EMR) would simplify migration of monitoring data
ACCEPTED MANUSCRIPT 10 into daily assessments. The monitoring data requested to include in the EMR includes vital signs trends, arrhythmia waveforms, and alarms information.
T
Participants stated that they are required to interact with the EMR multiple times
RI P
per day for patient record entry, so they prefer the EMR venue for data review in a single location.
SC
The Likert scale perceived usability of the central monitoring station was
NU
reported as moderately difficult, with a mean of 5.3 (SD 1.7) and a mode of 5 on the 10-point Likert scale. Only one participant had onsite training provided by the
MA
manufacturer when the equipment was installed several years prior. Two subjects stated that they had on-the-job training on using the monitoring
ED
equipment, but also reported that the training lasted 10 minutes and 15 minutes
PT
respectively. The remaining seven subjects had no training on the monitoring equipment. As part of the task analysis, it was noticed that none of the 138
CE
unique patients randomly selected for monitor setting manipulation had their
AC
settings adjusted from the manufacturer's factory default settings.
Subjective comments from participants were collected, including the response that all participants prefer to obtain information from the monitor at the patient bedside. Other comments were that a picture or video of the patient on the central station next to their physiological data would improve the clinicians' recall of that patient. A video would also help to determine the visual characteristics of a patient (e.g. whether they are moving, resting or in distress). Task Analysis Results
ACCEPTED MANUSCRIPT 11
1. Alarms Silencing
T
The alarm was silenced by use of keyboard by two subjects, and the other
RI P
8 subjects used the touchscreen. The mean time to silence the alarm measured from the time of the instruction to the absence of the audible alarm was 6.2
SC
seconds (SD 2.3). The perceived difficulty of this task was a mean of 8.9 (SD
NU
2.3) with a mode response rating of 10, implying that users believe that this is a very simple task. Of the 10 participants, only one knew the alarm type and
MA
patient whose alarm was silenced. No other participant could state the alarm type or patient with the alarm.
ED
2. Alarms and Waveform Review
PT
The perceived difficulty on the Likert scale was a mean of 8.0 (SD 3.2) and a mode of 10, rating user perception of performing these tasks as extremely
CE
simple. There was wide variability in task completion times for each task, and the
AC
task of displaying the ECG waveform from one-hour prior was particularly challenging for four subjects. There were subjective comments from users surrounding the themes of the timeline on the screen and that they are usable by touchscreen and thus requires the use of the mouse. Also, users commented that there is no intuitive method for seeing more than the current streaming ECG waveform. Participants also commented that clicking on the ECG waveform on the screen might be an intuitive method for obtaining more ECG waveform data. Comparison of the perceived simplicity of performing this function with the wide variation in task completion times and the number of button pushes suggests a
ACCEPTED MANUSCRIPT 12 lack of user insight into their ability to interact with the monitor interface. 3. Trends Display for Vital Signs
T
The perceived difficulty of displaying a 24-hour trend of heart rate was
RI P
rated as a mean of 7.8 (SD 2.9) with a response mode of 10. This task was achieved with a mean time of 28.6 seconds (SD 19.7) with 6.7 button pushes
SC
(SD 2.3). Participants were subsequently asked to display the trend of either the
NU
blood pressure (7 subjects) or pulse oximetry data (3 subjects) for another patient, taking a mean of 27.1 seconds (SD 30.3) with 7.8 button pushes (SD
MA
8.8). Performing these two tasks is very similar on the user interface evaluated, but the lack of difference in mean time between the first and second trend display
ED
suggests that no immediate intra-task learning occurred (p=0.91). Subjective
PT
comments about this task were centered on the lack of intuitiveness. Participants commented that clicking on the numerical heart rate, blood pressure, or pulse
CE
oximetry value should result in the display of the trend.
AC
4. Parameter Alarm Adjustment The perceived confidence of being able to perform this task was variable with a mean Likert scale score of 6.4 (SD 4.5) out of 10, and response mode of 10. Subjective comments provided by participants centered on automated adjustment of parameter alarms. The clinicians providing comments suggested that the monitor should automatically create a parameter range around the patient's vital signs and only alert clinicians of changes in the individualized measures for each patient. 5. Pacing Detection Settings
ACCEPTED MANUSCRIPT 13 The perceived confidence of being able to perform this task was variable with a mean Likert scale response of 4.7 (SD 5.1) with a mode of 10, and
T
performance took a mean of 18.4 seconds (SD 17.0) with 2.9 button pushes (SD
RI P
1.6). While conducting this task, participants were asked to distinguish whether they would activate the "Pace 1" or "Pace 2" settings that are offered in the user
SC
interface. There were two participants that knew that there is a difference
NU
between these two options, but neither of them could remember what the difference is. Subjects recommended manufacturers label them as to their actual
MA
function instead of Pace 1 and Pace 2. When asked what activation of pacing detection does for the monitoring system, only one participant provided a
ED
response stating, "The monitor puts a white spike on the ECG tracing".
PT
6. ST-segment Monitoring Practices All of the monitoring systems in this study are capable of continuous ST-
CE
segment monitoring. In this data, only the intensive care unit had activated any
AC
ST-segment monitoring on certain patients, while none of the monitors in the progressive care or emergency departments had any ST-segment monitoring activated. The perceived difficulty of these tasks was rated as moderately challenging on the zero to 10 Likert scale with a mean perceived difficulty of 5.9 (SD 4.6) and a mode of 10 by 2 subjects. Three of the 10 subjects did not complete all of the tasks due to reported difficulty and frustration. Adjusting the upper limit of ST-segment to 3mm above baseline, taking a mean of 25.8 seconds (SD 15.7) with 9.1 button pushes (SD 3.3). Subsequent adjustment of the ST-depression alarm to 3mm below the baseline in ECG lead
ACCEPTED MANUSCRIPT 14 III took a mean time of 27.0 seconds (SD 38.5) with 7.3 button pushes (SD 4.5). Consistent with current ECG monitoring guidelines, the participants were tasked
T
with adjusting the ST-segment alarms to 1mm above and 1mm below the
RI P
baseline in ECG lead V1, which took a mean time of 116.7 seconds (SD 18.6) and 20.6 button pushes (SD 6.5). The subjects relate part of the increase in time
SC
to completion of the final task in this domain to the identification of the ST-
NU
segment baseline. This represents the adjustment of only one monitoring lead to practice guidelines, and this may represent one of the challenges to instituting
MA
ST-segment monitoring.
Subjective responses by the three participants who did not complete the
ED
task included remarks about ST-segment monitoring being time consuming,
PT
creating too many monitoring alarms, and too time consuming to perform the settings. Of the participants that completed the task, all of them remarked that
CE
they felt that the monitor should be able to do this automatically, and that
AC
completing this task for relevant monitoring leads would just be too time consuming for all of the patients. 7. Respiratory Rate Monitoring by Impedance Participants were asked to use the central monitoring station to evaluate the respiratory waveform and to change the ECG monitoring lead for respiratory rate from lead II to lead I. Unknown to this team, changing leads for respiratory rate monitoring could only be done on bedside equipment. All participants were given the opportunity to perform the task on the bedside monitor, which all remarked that it was much easier to do this at the bedside. When asked about
ACCEPTED MANUSCRIPT 15 the use of impedance technology for respiratory rate monitoring, none of the participants were aware of how the monitor measured respiratory rate. Eight of
RI P
movement of the electrodes to measure respiratory rate”.
T
the ten participants stated that they believed that the monitor “detected
Discussion
SC
Consumer electronics have become popular due to their user-friendliness
NU
and intuitive designs. However, these characteristics are a result of large investments made to research and development in the area of human factors,
MA
human computer interaction and to the User Experience (UX) of the interface. In fact many electronic devices such as smart phones have a myriad of advanced
ED
features and yet general members of the public can adopt these technologies
PT
with ease and without the use of large instruction manuals2. This is mainly due to the implementation of UX principles such as affordance, i.e. the device should
CE
behave as expected or anticipated. For example, if the majority of users expect
AC
to see more electrocardiographic options when they click on the ECG signal – then this should be the case. Whilst popular electronic devices are dedicated to usability engineering, perhaps manufacturers of medical devices are a little behind. This has been recognized by regulatory bodies such as the FDA who now enforce medical devices to undergo a ‘usability validation’ before they are introduced into real clinical scenarios3,5. This is of great importance given suboptimal usability of medical devices can result in intragenic errors and a threat to patient safety. Limitations
ACCEPTED MANUSCRIPT 16 The main limitation of this experiment is the small sample size. However, according to an empirical function composed by usability experts, 80% of
T
usability issues can be uncovered by recruiting just five subjects6. Nevertheless,
RI P
we recognize that we would need more participants to gain a better representation of other quantitative data such as task completion times. Also, this
SC
study was conducted at a single teaching hospital using only one brand of
NU
monitoring equipment-limiting generalizability. These authors also recognize that 70% of the participants had no formal training in the use of these monitors, and
MA
although that is acceptable at this facility, it is not clear what training practices exist regarding monitoring equipment at other facilities warranting further
ED
research.
PT
Conclusions
Our impetus for embarking in this study was to gain insight into whether
CE
usability issues exist in hospital monitoring, and we conclude that physiological
AC
monitors should be improved regarding their clinician-friendliness and interaction design. This is evidenced by the fact that participants failed to complete a number of tasks and/or completed tasks within an extended amount of time. In addition, there was also a lack of confidence and competence in using a number of features of the monitor. This included a lack of proficiency in setting personalized thresholds for alarms. This illustrates that clinicians do not exploit the capability of medical machines. This fact should be considered when developing new functionalities for physiological monitors and more effort should be made to increase the uptake of current monitoring features.
ACCEPTED MANUSCRIPT 17 We also aimed to gather opinions and perceptions about physiologic monitors, and we found that monitor credibility is compromised by high false
T
alarm rates, and clinicians would prefer a single point of access for all patient
RI P
data--specifically full monitoring data available through the electronic health record. Multiple subjects cited this point as a barrier to incorporating monitoring
SC
data into clinical rounds and patient management. We recommend that hospitals
NU
enhance the competency of nurses in using physiological monitors through simulation-based training that would include scenarios where features such as
MA
ST-segment monitoring are of critical importance. Future work will include the recruitment of additional participants to validate task completion times. We will
ED
also consider video recording tasks for annotation purposes and for analyzing
James, J. 10 ways developers can meet user expectations and ease
AC
1.
CE
References
PT
specific events and usability issues.
frustrations. TechRepublic/US. 13 May 2009. Web. 7 July 2015. 2.
Monsees, J. Putting "intuitive" back into intuitive design. UX Magazine. 14
October 2014. Web 7 July 2015. 3. United States. Food and Drug Administration. Draft Guidance for Industry and Food and Drug Administration Staff-Applying Human Factors and Usability Engineering to Optimize Medical Device Design. Food and Drug Administration, 2011. http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDoc
ACCEPTED MANUSCRIPT 18 uments/ucm259748.htm 4. Likert, R. (1932). A Technique for the Measurement of Attitudes. Archives of
T
Psychology, 140, 1–55.
RI P
5. Drew, B., Califf, R., Funk, M., Kaufman, E., Krucoff, M., Laks, M., Macfarlane, P., Sommargren, C., Swiryn, S. & Van Hare, G. (2004). Practice Standards for
SC
Electrocardiographic Monitoring in Hospital Settings. Circulation; 110: 2721-
6.
NU
2746. doi: 10.1161/01.CIR.0000145144.56673.59.
FDA, Human Factors Implications of the New GMP Rule Overall
MA
Requirements of the New Quality System Regulation, 2010. Available: http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/PostmarketR
Nielsen, J. and Landauer, T.K. A mathematical model of the finding of
PT
7.
ED
equirements/Human Factors/ucm119215.htm (01.11.10).
usability problems. In Proceedings of INTERCHI 1993 (Amsterdam, the
AC
CE
Netherlands, Apr. 24–29). ACM Press, New York, 1993, 206–213.
ACCEPTED MANUSCRIPT 19 Table 1. Test Domains and Tasks Performed
SC
RI P
2. Alarms and waveform review.
MA
ED PT
AC
CE
5. Pacing Detection
NU
3. Trends Display for Vital Signs
4. Parameter Alarms Adjustment
6. ST-Segment Monitoring Practices
7. Respiratory Rate Monitoring by Impedance
1. Identify a patient with a monitoring alarm then silence the alarm (keyboard or touchscreen). 2. Ask participant to identify the patient or the alarm type or both. 1. Select a patient and display all of the arrhythmia alarms for the past 24-hours. 2. Display the ECG waveform from 1-hour ago. 3. Print a hard copy of multi-lead ECG for an ECG arrhythmia alarm condition. 1. Select a patient and display a 24-hour trend for heart rate (both ECG or pulse oximetry acceptable). 2. Display a 24-hour trend for either blood pressure or pulse oximetry. 1. Select a patient and increase the high heart rate alarm limit by 10 beats per minute. 2. Select another patient and decrease the low pulse oximetry alarm limit to 88%. 3. Select another patient and change the atrial fibrillation alarm from audible to an inaudible visual message. 1. Identify and select a patient that has a permanent or temporary pacemaker. 2. Activate the pacing detection feature for this patient (or verify it is active if already activated). 3. Ask participant to explain the difference between Pace 1 and Pace 2 functions. 1. Select a patient and change the STsegment elevation limit to 3mm in a single ECG lead. 2. Change the ST-depression limit to 3mm in a single ECG lead. 3. Consistent with current guidelines, adjust the ST-segment alarm thresholds to 1mm above and 1mm below the patient baseline in ECG lead V1. 1. Select a patient and adjust the respiratory rate detection lead from lead II to lead I.
T
1. Alarm Silencing
ACCEPTED MANUSCRIPT
Profession Physician
2
46
Male
Physician
3
47
Male
Physician
4 5 6 7 8 9 10
28 30 35 63 34 29 41 Mean 39.4
Female Female Male Female Female Female Female 70% Female
Nurse Nurse Nurse Nurse Nurse Nurse Nurse 70% Nurse
Range 28-63
AC
CE
PT
RI P
SC
NU
ED
SD 11.6
T
Table. 2 Sample Characteristics Subject Age (years) Gender 1 37 Male
MA
20
Specialty Pulmonary Critical Care Anesthesia Critical Care Pulmonary Critical Care Telemetry Telemetry Telemetry Telemetry Emergency Critical Care Critical Care 3 Critical Care Physicians 4 Telemetry Nurses 3 Emergency or Critical Care Nurses
ACCEPTED MANUSCRIPT 21 Table. 3 Clinician Perceptions of Monitoring Usability
Mean (SD) Mode
1. Alarm Silencing 2. Alarms and Waveform Review 3. Trends Display for Vital Signs 4. Parameter Alarm Adjustment 5. Pacing Detection Settings 6. ST-Segment Monitoring Practices 7. Respiratory Rate Monitoring by Impedance 8. Overall perceived usability of central monitoring station.
8.9 (2.3) 8.0 (3.2) 7.8 (2.9) 6.4 (4.5) 4.7 (5.1) 5.9 (4.6) Insufficient data 5.3 (1.7)
AC
CE
PT
ED
MA
NU
SC
RI P
T
Perceived Task Difficulty
10 10 10 10 10 10 5
ACCEPTED MANUSCRIPT 22 Table 4. Task Analysis Data Summary.
Review ECG Alarms for past 24h
Display ECG waveform from 1hr ago
PT
Display HR Trend past 24h
CE
3. Trends Display for Vital Signs
AC
Display 24h Trend of Blood Pressure or Pulse Oximetry
4. Parameter Alarms Adjustment
Increase high heart rate alarm by 10 beats/min
Decrease low pulse oximetry alarm to 88%
Change atrial fibrillation to messaging
5. Pacing
(1-17.2) 2.3 20.3
1 0 5.9
(8.1-47.6) 14.7 81.4
(14-150.9) 52.3 34.4
Activate pacing
T
1
ED
Print multi-lead ECG hardcopy for an arrhythmia alarm
6.2
Comments
RI P
2. Alarms and Waveform Review
Range SD
Mean Number of Button Pushes Range SD
SC
Silence an alarm
Mean Time (seconds)
NU
1. Alarm Silencing
Task Performed
MA
Test Domain
(2-17) 4.6 15
(5-36) 9.3 8.2
(12.6-75.5) 20.0 28.6
(3-25) 6.6 6.7
(12.7-75.4) 19.7 27.1
(4-12) 2.3 7.8
(4-94.9) 30.3 15.3
(2-26) 8.8 6.3
(6.1-24.3) 5.4 28.1
(4-8) 1.2 9.3
(10.2-109.8) 33.4 16.9
(3-26) 7.2 6.0
(6-31) 8.6 18.4
(2-9) 2.1 4.9
One subject unable to complete after 140 seconds
ACCEPTED MANUSCRIPT 23 Detection
detection
Change ST Elevation Alarm to 3mm
(3-8) 1.6 9.1
(16.5-62.3) 15.7 27.0
(5-13) 3.3 7.3
Change respiratory rate lead from II to I
AC
CE
PT
7. Respiratory Rate Monitoring
SC
NU
(97.8-138.0) 18.6 Unable to complete from central station
ED
Change STSegment Alarms to Practice Guideline
(7.8-122.6) 38.9 116.7
MA
Change ST Depression Alarm to -3mm
RI P
T
6. ST-Segment Monitoring Practices
(6.1-48.7) 17.0 25.8
(5-18) 4.5 20.6
(12-30) 6.5
One subject unable to complete any of these tasks; two more subjects were unable to complete task 3 in this domain