Usability evaluation of a comprehensive national health information system: relationship of quality components to users’ characteristics

Journal Pre-proof Usability evaluation of a comprehensive national health information system: relationship of quality components to users’ characteristics Fateme Rangraz Jeddi, Ehsan Nabovati, Reyhane Bigham, Reza Khajouei

PII:

S1386-5056(18)31024-4

DOI:

https://doi.org/10.1016/j.ijmedinf.2019.104026

Reference:

IJB 104026

To appear in:

International Journal of Medical Informatics

Received Date:

9 September 2018

Revised Date:

23 October 2018

Accepted Date:

28 October 2019

Please cite this article as: Rangraz Jeddi F, Nabovati E, Bigham R, Khajouei R, Usability evaluation of a comprehensive national health information system: relationship of quality components to users’ characteristics, International Journal of Medical Informatics (2019), doi: https://doi.org/10.1016/j.ijmedinf.2019.104026

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Published by Elsevier.

Usability evaluation of a comprehensive national health information system: relationship of quality components to users’ characteristics

Fateme Rangraz Jeddi 1, 2, Ehsan Nabovati 1, 2, *, Reyhane Bigham 2, 3, Reza Khajouei 4 1

ro of

Health Information Management Research Center, Kashan University of Medical Sciences, Kashan, Iran 2 Department of Health Information Management & Technology, School of Allied Health Professions, Kashan University of Medical Sciences, Kashan, Iran 3 Student research committee, Kashan University of Medical Sciences, Kashan, Iran 4 Medical Informatics Research Center, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Iran

Health Information Management Research Center

-p

* Corresponding Author: Ehsan Nabovati, Health Information Management Research Center, Kashan University of Medical Sciences, Kashan, IR Iran. E-mail: [email protected] Address:

re

Pezeshk Blvd, 5th of Qotbe Ravandi Blvd - Pardis Daneshgah, School of Allied Health Professions, Kashan University of Medical Sciences, Kashan, IRAN

na

Summary points

lP

Postal Code: 8715973449, Tel: 00983155548883, Fax: 00983155548883

What was already known on the topic: Poor usability reduces the use of information systems and hinders their acceptance by

ur



the users.

Usability is a context-dependent feature of information systems which can be associated

Jo



with users’ characteristics.

What this study adds: 

A national health information system which is used by a large number of users across a developing country have low usability.



Learnability of information systems have significant relationships with the user’s position, education level, and field of education. 1



Physicians have assigned significantly lower scores for system efficiency, helpfulness and global usability, compared to other users.



Users' practice experience and age have significant linear inverse relationships with

ro of

efficiency, helpfulness, and learnability.

Abstract

Objective: One of the most important methods for evaluating information systems is

-p

usability evaluation. Usability is a context-dependent qualitative feature that is measured by multiple quality components that can be related to users’ characteristics. This study was

re

conducted to evaluate the usability of a comprehensive national health information system

lP

(SIB; an abbreviation for the Persian equivalent of ‘integrated health system’) from the perspective of different users and to determine the relationship between quality components

na

and users’ characteristics.

Method: The study population were users of the national health information system (n= 309)

ur

at health centers and health homes affiliated to Kashan University of Medical Sciences, Kashan, Iran. Data were collected using Software Usability Measurement Inventory (SUMI)

Jo

questionnaire which measures users' experiences of software interface in five quality components (i.e. affect, efficiency, helpfulness, control, and learnability) and provides a global usability score. SUMI scores were analyzed according to an extensive reference database (SUMISCO). The relationships between quality components and users’ characteristics were investigated by one-way analysis of variance (ANOVA), independent ttest, and Pearson’s correlation coefficient. 2

Results: A total of 250 users completed the questionnaire (response rate= 81%). The mean scores of all quality components were significantly lower than the mean of SUMISCO. Learnability score had significant relationships with the user’s position, education level, and field of education (P<0.001). Physicians scored significantly lower than other users in efficiency, helpfulness and global usability (P<0.05). Users' practice experience and age had significant linear inverse relationships with efficiency, helpfulness, and learnability (P<0.05). Conclusions: The national health information system which is used by a large number of

ro of

users across a developing country have low usability. Given the significant relationships

between the users’ characteristics of such systems and quality components, it is essential to

consider the characteristics and needs of various user groups during the processes of system

-p

analysis and design.

re

Key words: Health Information Systems, Evaluation Studies, User-Computer Interface,

Jo

ur

na

lP

Usability

3

1. Introduction Information systems are extensively used in various healthcare settings and have improved the quality, efficiency, and effectiveness of health services and the overall patient satisfaction [1, 2]. In line with the reform of health system in Iran ("Iran’s health system reform plan 2014") [3], a comprehensive integrated national health information system (SIB; an abbreviation for the Persian equivalent of "integrated health system”) was implemented in 2015 to provide integrated public health services, provide the requirements for the

ro of

implementation of the referral system, increase the accessibility of the reports on public

health, and ultimately improve the quality of health services. SIB was implemented and used in health homes and (rural and urban) health centers in Iran. Theses settings have close contact with

-p

people in society and provide the first level of health services for the entire population. All healthrelated information of population gathered during provision of primary health services are

re

registered in SIB. In subsequent referrals of care recipients, this information is available for providers

lP

through SIB. Since the inception of this reform until June 2018, electronic health records have

been created for over 75 million people in Iran (more than 90 percent of the population). The

across the country.

na

users of this system are about 100,000 healthcare providers from more than 30,000 settings

ur

The effectiveness of information systems depends on their quality which is determined by characteristics such as functionality, reliability, usability, and portability [4, 5]. Usability is

Jo

one of the key features of the information system quality defined by the ease of use and the extent to which a product is used by specific users for specific goals with effectiveness, efficiency, and satisfaction [6]. Poor usability reduces the use of information systems and hinders their acceptance by the users [7]. This will, in turn, decrease the confidence in information systems and reduce satisfaction, effectiveness, and efficiency [8].

4

The Institute of Medicine reported [9], the poor usability of the electronic health record (EHR) in the US in 2012. According to studies in Finland, Denmark, and Canada [10, 11], health professionals reported the poor usability of the EHR during 2010-14 and its negative effects on their satisfaction and efficiency. Given the extensive use of the national health information system in healthcare settings in Iran, its poor usability would have negative effects on the acceptance, satisfaction, and confidence of a large number of users. Public health plans based on reports of this information system will consequently be disrupted and

ro of

indirect adverse public health effects will be performed. According to ISO9421-11 [6], usability is a context-dependent feature of information systems which can be associated with users’ characteristics. Some studies have shown a relationship

-p

between usability of an information system (and any of its components, e.g. satisfaction,

effectiveness, efficiency) and the users’ characteristics; however, some other studies have

re

failed to confirm such a relationship. In a recent study, Reis et al. [12] assessed the usability

lP

of Idaho Health Data Exchange system from the perspective of three groups of users (healthcare professionals, licensed health professionals, and staff members) and reported that

na

usability of a system is different from the perspective of different users. Another study showed that age, gender, and information technology skills of users affected their performance in using the information system, but education level had no effects in this regard

ur

[13]. Moreover, multiple studies have reported higher satisfaction with and tendency to use

Jo

information systems among younger physicians than in their older counterparts [14-16]. In contrast, a study did not detect any relationships between demographic features of physicians and nurses and their satisfaction with the computerized physician order entry (CPOE) system [17]. Given the extensive use of SIB by various end-users in Iran, it is necessary to evaluate its usability from the perspective of its end-users. Moreover, considering the diversity of users’ 5

characteristics (e.g. position, practice experience, education level, and field of education), the relationships between usability of SIB and users’ characteristics can be determined. To the best of our knowledge, no study has yet evaluated the usability of SIB. As the first phase of a project to evaluate the usability of SIB, this study was performed to measure end-users’ usability experiences of SIB and determine the relationships between usability components and users’ characteristics.

ro of

2. Methods This descriptive cross-sectional study was conducted in Iran in 2017. The study population

consisted of all SIB end-users (n=309) in (rural and urban) health centers and health homes affiliated to Kashan University of Medical Sciences, Kashan, Iran. These settings cover a

-p

population of 400,000 people in central Iran. At the time of the study, SIB had already been

re

implemented in 96 health centers and health homes around the city of Kashan and 535 user accounts were created. After combining the accounts of various users holding different

lP

positions in a number of settings, 309 unique end-users were determined. All the end-users had the same experience with SIB from its implementation date.” Initially, the user interface

na

of different parts of the system was inspected. Although users were different based on their position and field of education, the overall design of user interface and the method of

ur

interaction with the different modules of the system were mostly the same.

Jo

Data were collected using the Software Usability Measurement Inventory (SUMI) questionnaire [18] which measures users' experiences of working with the software interface in five quality components (i.e. affect, efficiency, helpfulness, control, and learnability) and provides a global usability score. This questionnaire has been translated into 20 different languages (including Persian) and its validity and reliability have been previously confirmed [19]. The validity and reliability of the Persian version were also confirmed in another study [20]. Several studies have used the SUMI for evaluating the usability of information systems 6

from the perspective of end-users [12, 21-25]. Data analysis, report generation, and SUMI support at the international level are carried out based on an extensive reference database (including 2000+ users’ responses about the usability of various information systems) and using an analysis and reporting tool called SUMISCO [18, 19]. The SUMISCO report determines whether the usability of an information system is higher or lower than the mean usability of the reference database. The SUMI contains 50 three-option (agree, undecided, and disagree) items measuring five quality components. The global subscale contains 25

ro of

items selected from all questionnaire items (Table 1). Higher scores of a subscale indicate the more positive attitude of users toward it [19]. One of the researchers administered the

questionnaires to the end-users and asked individuals who were willing to participate to

-p

complete the questionnaire. Necessary explanations about the study objectives were provided. The users who agreed to participate in the survey were asked to complete it anonymously.

re

The participants were also assured of the confidentiality of their personal information. After initial administration of questionnaires, the researcher made two visits to collect completed

lP

questionnaires (10 and 20 days after administration).

na

Table 1. SUMI subscales and their definitions [19]. Quality component Definition

Affect

ur

(SUMI subscale)

This indicates the user's general emotional reaction to the software.

Jo

High affect reflects users who find using the software satisfying and

Efficiency

interesting. Which is the relation between (a) the accuracy and completeness with which users achieve certain goals and (b) the resources expended in achieving them.

7

Helpfulness

The degree to which the software is self-explanatory, as well as the adequacy of help facilities and documentation.

Control

The extent to which the user feels in control of the software, as opposed to being controlled by the software, when carrying out the task.

Learnability

The speed and facility with which the user feels that they have been able to master the system, or to learn how to use new features when necessary. This is a kind of bottom-line composite measure of usability that

ro of

Global

describes the user’s generalized perceived quality of use.

-p

Analysis

re

Categorical variables were described using frequencies and percentages with 95% confidence intervals (CIs). Data from SUMI subscales were presented as T-scores and analyzed by

lP

SUMISCO software [19]. A T-score is a standardized score computed according to the general distribution of scores. The scores were rescaled to obtain a mean value of 50 and a

na

standard deviation of 10. Between-group differences in the mean scores of SUMI subscales were evaluated using one-way analysis of variance (ANOVA) for categorical variables and

ur

independent t-tests for binomial qualitative variables. The relationships between scores and quantitative variables were determined by Pearson’s correlation analysis. All statistical tests

Jo

were performed using SPSS 16.0 (SPSS Inc., Chicago, IL, USA) at a significant level of 0.05.

3. Results

A total of 309 questionnaires were distributed in 96 health centers and health homes affiliated to Kashan University of Medical Sciences. However, 250 completed questionnaires were

8

finally returned (response rate: 81 percent). The respondents’ demographic characteristics are presented in Table 2. Table 2. Demographic characteristics of the respondents (n = 250). Characteristic

Mean ± SD

Age (year)

34.3 ± 7.4

N (%)

Gender 202 (80.8)

Male

48 (19.2)

ro of

Female

Position

7 (2.8)

Mental health expert

12 (4.8)

-p

Nutrition expert

Physician

13 (5.2) 55 (22)

re

Healthcare workers *

55 (22)

Healthcare providers **

Education level

108 (43.2)

9.8 ± 8.2

na

Practice experience (year)

lP

Midwife

9 (3.6)

ur

High school education

43 (17.2)

Graduate diploma

40 (16)

Jo

High school diploma

Bachelor of science

134 (53.6)

Master’s degree

11 (4.4)

General Medicine

13 (5.2)

Field of education Nutrition

7 (2.8)

9

Clinical psychology

12 (4.8)

Medicine

13 (5.2)

Family health

20 (8)

Personal caregiver

54 (21.6)

Midwifery

62 (24.8)

Public health

82 (32.8)

* Healthcare workers: Users with high school education or a high school diploma working in health homes.

ro of

** Healthcare providers: Users holding a bachelor’s degree working in urban health centers.

Healthcare providers and workers (in health homes and rural and urban health

-p

centers) have an in-depth knowledge about the health of their community. They

identify health-related issues, collect health data, and discuss health concerns with

re

the people. They provide referrals for health needs such as nutrition, education, and

lP

mental health services.

Figure 1 shows the mean scores of SUMI subscales. These scores were normalized according

na

to the mean value of 50 for the reference database (SUMISCO). As seen in Figure 1, the obtained scores for all subscales were significantly lower than the mean value of the

Jo

ur

reference database.

10

ro of

-p

Figure 1. SUMI subscale scores (means with 95% CIs)

The scores of subscales based on users' gender, position, education level, and field of

re

education are mentioned in Appendix 1. The efficiency, helpfulness, and global usability

lP

scores showed significant relationships with users' position and field of education. In fact, individuals who held a position as mental health expert and those who had studied clinical psychology had the highest scores in these three subscales. Holding the position of a

na

physician and studying medicine were associated with the lowest scores in these subscales.

ur

Learnability score had significant relationships with users' position, education level, and field of education. Mental health experts, individuals with a master’s degree, and those who had

Jo

studied clinical psychology had the highest learnability scores. Meanwhile, healthcare workers and individuals with high school education had the lowest scores. Table 3 shows Pearson’s correlations between the scores of SUMI subscales and the respondents' practice experience and age. As seen, the scores of efficiency, helpfulness, and learnability had significant linear inverse relationships with users' practice experience and age

11

(i.e. the scores of these subscales were significantly lower in individuals with higher practice experience and age). Table 3. The relationships between quality components (SUMI subscales) and the respondents' practice experience and age. Pearson

Efficiency

Affect

Helpfulness

Control

Learnability

correlation

Age

Usability

R

-.17

.00

-.12

-.04

-.27

-.10

p value

.00*

.99

.04*

.50

.00*

.11

R

-.17

.00

-.13

p value

.00*

.95

.03*

-.05

-.22

-.10

.42

.00*

.09

Jo

ur

na

lP

re

-p

* Difference significant at p<0.05

ro of

Practice experience

Global

12

4. Discussion The results of this study indicated low scores of usability in all quality components for the evaluated system. These quality scores were significantly lower than the mean scores of the reference database (SUMISCO). System learnability was significantly lower in less educated users (i.e. high school education, high school diploma) than in those with higher education levels (i.e. master’s degree, general medicine). Moreover, physicians had significantly lower efficiency, helpfulness, and global usability scores than other users. The users’ efficiency,

ro of

helpfulness, and learnability significantly reduced with aging and increased practice experience.

From the perspective of users, the integrated national health information system, which is an

-p

EHR system used by more than 100,000 users in a developing country, had poor usability. A study conducted in 2014 reported that despite the adoption of EHRs in Finland, Denmark,

re

and Canada, these systems had poor usability [11]. Another study investigated physicians’

lP

experiences of working with the EHR in Finland during 2010 and 2014 concluded that the system had poor usability and this quality criteria did not improve during these two years

na

[10]. Similarly, in 2012, the Institute of Medicine reported that despite the extensive use of EHRs, they had poor usability in the U.S. [26]. To overcome the poor usability of EHRs, the

ur

U.S. Department of Health and Human Services’ Office of the National Coordinator for Health Information Technology (ONC) developed certification requirements for usability and

Jo

obliged EHR developers to comply with them [27]. Given the results of this study and similar research (indicating the poor usability of EHRs), countries intending to implement national health information systems are recommended to first develop usability requirements for these systems and oblige developers to comply with them. Besides that, developers should note that using user-centered design methods can improve the usability of information systems [27].

13

The results of this study showed significantly lower system learnability among poorly educated users (i.e. high school education and high school diploma) compared to more educated users (i.e. master’s degree and general medicine). Users' education level is a human factor with a huge effect on users' adoption of information systems [28-30]. A study showed that more educated personnel used computers significantly more than others [31]. These results indicate that less educated users need more training and support for better interaction with information systems.

ro of

According to our findings, physicians’ scores in efficiency and helpfulness were significantly lower than those of other users. Since efficiency is measured by time, accuracy, and

completeness [32], the efficiency of information systems relates to the time spent by a user to

-p

accurately complete an activity on the information system. Physicians’ low efficiency score in present study show that, performing some activities on the information system is time-

re

consuming for them and/or they cannot carry out some activities accurately. The low

lP

helpfulness score of the physicians indicate that from their perspective, the information system was not sufficiently self-exploratory and lacked adequate "Help" functionality and

na

documentation tools. These results concur with studies that reported poor usability and timeconsuming activities (especially data entry) are the most important reasons for physicians' reluctant to adopt and dissatisfaction with information systems [33-35]. It has been shown

ur

that physicians were confused and concerned about the time taken for each activity when

Jo

working with EHR [34, 36]. A significant proportion of the time spent by physicians to interact with information systems is devoted to data entry and if this problem is resolved, they would electronically register all data [37]. Design or redesign of information systems according to user needs, especially for physicians who are pressed for time, can increase users’ efficiency and helpfulness. Addition of "Help" functionality can also contribute to better performance of the users.

14

The results of this study showed that scores of efficiency, helpfulness, and learnability significantly decreased with increased age and practice experience of the users. These results indicated that older and highly experienced users believed that they spent a lot of time performing activities on the system, the system lacked adequate "Help" functionality, and the learnability of the system is not adequate. Previous studies have reported conflicting results about the relationship between users' age and their views on information systems’ ease of use. In agreement with the present study, a study showed that, in comparison with younger nurses,

ro of

older ones had lower technical skills and higher reluctance to use information systems [38]. Other studies also confirmed that older people were less willing, efficient, and happy to use modern information technologies, such as EHR, than younger people [13, 16, 39, 40]. In

contrast, Khajouei et al. [17] found no significant relationships between physicians' age and

-p

their attitude toward the information systems’ ease of use. Moreover, that study showed that

re

nurses' satisfaction with the CPOE system increased with increasing age. Such inconsistency between the results of this last study and others can be justified by the fact that a long time

lP

(over a decade) has elapsed since the establishment of the information system evaluated in the last study and their users have become fully familiar with the system at the time of

na

evaluation. Due to the rapid developments in the field of information technology, older users seem to be less knowledgeable and skilled in this area compared to younger users and may

ur

have greater difficulties with these technologies [41, 42]. Therefore, the characteristics and limitations of older users should be more considered during the design phase of new health

Jo

information systems. Providing these users with ongoing support and explanations about the advantages of these information systems can improve their learnability. Moreover, for greater efficiency, attention should be paid to the ease of use of these systems. Strengths and limitations

15

Compared to other usability evaluation methods, using a questionnaire facilitated the collection of experiences and views of a greater number of end users who better represented the study population. Moreover, using anonymous questionnaires further enhanced the perceived impersonality of the respondents compared to other data collection methods such as interviews, observation, and user testing [43]. The present study had some limitations. First, although the study was performed in a relatively populated region in the center of Iran, the studied population may not represent the

ro of

entire population of the country. Second, although the initial inspection showed that the overall design of user interface and the method of interaction with the different modules of the system were mostly the same, there may be a limitation due to different experiences of

-p

multiple user groups with the system. Third, effectiveness and satisfaction proposed by ISO as components of usability were not included in the questionnaire under a specific title.

re

However, items related to these components were assessed under other SUMI subscales. For instance, items "The instructions and prompts are helpful" and "The software is fast enough"

lP

evaluated effectiveness and items "I enjoyed the time I spent using this software" and "Working with this software is satisfying" in the questionnaire evaluated satisfaction. In this

na

study to maintain the framework of the SUMI, subscales suggested by its developers were used in the analysis. Other users of this questionnaire can make changes if necessary and use

ur

other categories matching usability components. The last limitation was the difficulty of completing the questionnaire due to both the relatively large number of questions and the

Jo

large number of respondents in health centers. To resolve this limitation, the researchers tried to present users with the questionnaires with prior arrangement and during quiet working hours and to encourage users to complete the questionnaires by explaining the importance of results of the research. Those users failing to complete the questionnaires during the first round were followed up in the second round. Implications of the study 16

Given the low score of global usability of the evaluated system, future studies are recommended to identify the causes of usability problems using other usability evaluation methods such as "think aloud". The low score of global usability indicate that the universal use of an information system across a country does not confirm its high usability and the system may have usability problems. Moreover, in order to increase the efficiency and usability of information systems designed to be utilized by different users with different characteristics (e.g. education level and age), developers should meet the requirements and

ro of

needs of all groups. Since the results of the present study and similar ones indicated the lower learnability of information systems among less educated and older users, greater attention should be paid to training and support of this type of users.

-p

5. Conclusion

re

The integrated national health information system, which is used by a large number of users across a developing country, had low usability. It is necessary for countries intending to

lP

establish national health information systems to develop usability requirements and oblige developers to comply with them. Since the users’ characteristics (e.g. position, education

na

level, work experience and age) of these systems have significant relationships with usability components, the needs of different user groups should be considered during the design

ur

process. Finally, the training and support of older and less educated users and simplifying the

Jo

activities on these information systems for physicians are recommended.

Conflict of interests The authors have no conflict of interests to declare.

Funding 17

This study was supported by a grant from Kashan University of Medical Sciences Research Council (Number: 96049)

Acknowledgments Healthcare providers and workers at health centers and health homes affiliated to Kashan

Jo

ur

na

lP

re

-p

ro of

University of Medical Sciences are thanked for their participation in the survey.

18

References

Jo

ur

na

lP

re

-p

ro of

1. Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health affairs. 2011;30(3):464-71. 2. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Annals of internal medicine. 2006;144(10):742-52. 3. Esmailzadeh H, Rajabi F, Rostamigooran N, Majdzadeh R. Iran Health System Reform Plan Methodology. Iranian Journal of Public Health. 2013;42(Supple1):13-7. 4. Bevan N. International standards for HCI and usability. International journal of humancomputer studies. 2001;55(4):533-52. 5. Haux R, Winter A, Ammenwerth E, Brigl B. Strategic Information Management in Hospitals: An Introduction to Hospital Information Systems: Springer; 2004. 6. ISO 9241-11:1998. Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability 1998 [Available from: https://www.iso.org/standard/16883.html. 7. Cresswell K, Sheikh A. Organizational issues in the implementation and adoption of health information technology innovations: an interpretative review. International journal of medical informatics. 2013;82(5):e73-86. 8. Khajouei R, de JONGH D, Jaspers MW, editors. Usability evaluation of a computerized physician order entry for medication ordering. MIE; 2009. 9. Institute of Medicine: Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: The National Academies Press; 2012. 234 p. 10. Kaipio J, Lääveri T, Hyppönen H, Vainiomäki S, Reponen J, Kushniruk A, et al. Usability problems do not heal by themselves: National survey on physicians’ experiences with EHRs in Finland. International journal of medical informatics. 2017;97:266-81. 11. Kushniruk A, Kaipio J, Nieminen M, Hyppönen H, Lääveri T, Nohr C, et al. Human Factors in the Large: Experiences from Denmark, Finland and Canada in Moving Towards Regional and National Evaluations of Health Information System Usability: Contribution of the IMIA Human Factors Working Group. Yearbook of medical informatics. 2014;9(1):67. 12. Reis J, MacKenzie L, Soelberg T, Smith J. Assessment of the usability and impact of the Idaho Health Data Exchange (IHDE). Journal of medical systems. 2016;40(4):102. 13. Georgsson M, Staggers N. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. Journal of the American Medical Informatics Association : JAMIA. 2016;23(1):5-11. 14. Decker SL, Jamoom EW, Sisk JE. Physicians in nonprimary care and small practices and those age 55 and older lag in adopting electronic health record systems. Health Affairs. 2012;31(5):110814. 15. DesRoches CM, Campbell EG, Rao SR, Donelan K, Ferris TG, Jha A, et al. Electronic health records in ambulatory care—a national survey of physicians. New England Journal of Medicine. 2008;359(1):50-60. 16. Raglan GB, Margolis B, Paulus RA, Schulkin J. Electronic health record adoption among obstetrician/gynecologists in the United States: physician practices and satisfaction. Journal for Healthcare Quality. 2014. 17. Khajouei R, Wierenga PC, Hasman A, Jaspers MWM. Clinicians satisfaction with CPOE ease of use and effect on clinicians’ workflow, efficiency and medication safety. International journal of medical informatics. 2011;80(5):297-309. 18. Kirakowski J, Corbett M. SUMI: The software usability measurement inventory. British journal of educational technology. 1993;24(3):210-2. 19. Kirakowski J. Background notes on the SUMI questionnaire 2018 [cited 2018 01/15/2018]. Available from: http://sumi.uxp.ie/about/sumipapp.html. 19

ro of

20. AZIZI R, ZAKERIAN S, RAHGOZAR M. Determining Reliability and Validity of the Persian Version of Software Usability Measurements Inventory (SUMI) Questionnaire. International Journal of Occupational Hygiene. 2015;5(1):31-4. 21. Khajouei R, Azizi A, Atashi A. Usability Evaluation of an Emergency Information System: A Heuristic Evaluation. J Health Admin. 2013;16(52):61-72. 22. Kirakowski J, Claridge N. Taking the measure of user-centered design. Cutter IT Journal. 2003;16(10):21-7. 23. Laugwitz B, Held T, Schrepp M, editors. Construction and evaluation of a user experience questionnaire. Symposium of the Austrian HCI and Usability Engineering Group; 2008: Springer. 24. Nunes I, Kirakowski J. Interfaces Usability-Portuguese version of the Software Usability Measurement Inventory (SUMI). 2010. 25. O'Malley G, Dowdall G, Burls A, Perry IJ, Curran N. Exploring the usability of a mobile app for adolescent obesity management. JMIR mHealth and uHealth. 2014;2(2). 26. Committee on Patient Safety, Health Information Technology, Institute of Medicine, Health IT and Patient Safety: Building Safer Systems for Better Care. Washington (DC): National Academies Press (US)

Jo

ur

na

lP

re

-p

Copyright 2012 by the National Academy of Sciences. All rights reserved.; 2011. 27. Health information technology: standards, implementation specifications, and certification criteria for electronic health record technology, 2014 edition; revisions to the permanent certification program for health information technology. Final rule. Federal register. 2012;77(171):54163-292. 28. Farzandipur M, Jeddi FR, Azimi E. Factors Affecting Successful Implementation of Hospital Information Systems. Acta informatica medica : AIM : journal of the Society for Medical Informatics of Bosnia & Herzegovina : casopis Drustva za medicinsku informatiku BiH. 2016;24(1):51-5. 29. Handayani PW, Hidayanto AN, Budi I. User acceptance factors of hospital information systems and related technologies: Systematic review. Informatics for health & social care. 2017:1-26. 30. Handayani PW, Hidayanto AN, Pinem AA, Hapsari IC, Sandhyaduhita PI, Budi I. Acceptance model of a Hospital Information System. International journal of medical informatics. 2017;99:11-28. 31. Al-Gahtani SS. Computer technology acceptance success factors in Saudi Arabia: an exploratory study. Journal of Global Information Technology Management. 2004;7(1):5-29. 32. Frøkjær E, Hertzum M, Hornbæk K, editors. Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? Proceedings of the SIGCHI conference on Human Factors in Computing Systems; 2000: ACM. 33. Boonstra A, Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC health services research. 2010;10(1):231. 34. Friedberg MW, Chen PG, Van Busum KR, Aunon FM, Pham C. Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy: Rand Corporation; 2013. 35. Meade B, Buckley D, Boland M. What factors affect the use of electronic patient records by Irish GPs? International journal of medical informatics. 2009;78(8):551-8. 36. Likourezos A, Chalfin DB, Murphy DG, Sommer B, Darcy K, Davidson SJ. Physician and nurse satisfaction with an electronic medical record system. The Journal of emergency medicine. 2004;27(4):419-24. 37. Payne TH, TenBroek AE, Fletcher GS, Labuguen MC. Transition from paper to electronic inpatient physician notes. Journal of the American Medical Informatics Association. 2010;17(1):10811. 38. Kummer T-F, Schäfer K, Todorova N. Acceptance of hospital nurses toward sensor-based medication systems: a questionnaire survey. International journal of nursing studies. 2013;50(4):508-17.

20

Jo

ur

na

lP

re

-p

ro of

39. Nabovati E, Vakili-Arki H, Taherzadeh Z, Saberi MR, Abu-Hanna A, Eslami S. A survey of attitudes, practices, and knowledge regarding drug–drug interactions among medical residents in Iran. International Journal of Clinical Pharmacy. 2017:1-9. 40. Olson KE, O’Brien MA, Rogers WA, Charness N. Diffusion of technology: frequency of use for younger and older adults. Ageing international. 2011;36(1):123-45. 41. Kang NE, Yoon WC. Age- and experience-related user behavior differences in the use of complicated electronic devices. International Journal of Human-Computer Studies. 2008;66(6):42537. 42. Laberge JC, Scialfa CT. Predictors of Web Navigation Performance in a Life Span Sample of Adults. Human Factors. 2005;47(2):289-302. 43. Bowling A. Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health. 2005;27(3):281-91.

21

Appendix 1. The relationships between quality components (SUMI subscales) and respondents' characteristics. SUMI subscale score (mean ± SD) Respondents' Efficiency

Affect

Helpfulness

Control

Learnability

Global

characteristics Usability

Nutrition expert

43.43±17.72

41.14±19.75

43.71±8.53

41.43±9.50

54.71±9.37

42.71±15.22

Mental health

46.25±11.25

46.42±13.49

50.33±6.52

45.17±10.93

60.33±9.69

46.67±10.45

Physician

34.15±16.91

39.23±17.44

38.54±15.96

38.85±12.80

50.92±12.09

35.46±14.08

Healthcare

37.82±16.12

43.53±17.66

47.02±13.38

42.67±12.92

38.84±15.71

42.60±14.27

Midwife

35.35±13.83

36.49±15.98

41.95±12.76

41.27±12.42

46.75±12.91

37.76±12.15

Healthcare

40.92±14.89

44.04±16.21

-p

Position

47.44±13.37

49.92±13.79

44.28±13.24

0.04*

0.09

0.5

0.00*

0.02*

41.11±12.37

49±14.67

53.33±14.05

48.33±10.07

36.67±15.51

48.33±12.32

44.19±13.85

41.05±13.19

39.28±14.31

40.35±14.21

43.15±17.45

46.02±14.99

44.42±12.19

45.80±14.19

42.75±14.23

39.84±14.69

41.24±15.94

45.67±12.10

42.49±11.11

49.95±13.81

42.02±12.82

Master’s degree

49.64±11.57

48.64±16.52

53.18±9.25

48.18±12.12

60.09±10.02

50±12.30

General

34.15±16.91

39.23±17.44

38.54±15.96

38.85±12.80

50.92±12.09

35.46±14.08

0.07

0.5

0.05

0.18

0.00*

0.08

ro of

expert

worker

p value1

education High school

35.37±15.51

Graduate

Bachelor of science

38.15±16.03

Jo

diploma

41.26±18.50

ur

diploma

na

education High school

0.03*

lP

Level of

re

provider

43.99±11.06

medicine p value1

22

Field of education Nutrition

43.43±17.72

41.14±19.75

43.71±8.53

41.43±9.50

54.71±9.37

42.71±15.22

Clinical

46.25±11.25

46.42±13.49

50.33±6.52

45.17±10.93

60.33±9.69

46.67±10.45

Medicine

34.15±16.91

39.23±17.44

38.54±15.96

38.85±12.80

50.92±12.09

35.46±14.08

Family health

34.65±14.71

43.35±16.76

44.80±15.16

43.45±11.18

41.15±17.19

41.15±13.29

Personal

36.52±15.62

41.87±17.57

45.61±14.09

41.57±13.07

39.96±14.05

41.43±14.21

Midwifery

36.03±14.23

36.82±16.31

42.65±13.05

41.50±12.31

46.32±13.82

38.52±12.84

Public health

43.22±14.57

45.68±15.82

48.94±12.19

44.89±10.82

51.77±13.29

45.78±12.78

p value1

0.01*

0.07

0.03*

0.4

0.00*

0.01*

Female

38.36±14.90

40.72±16.60

45.30±13.23

42.30±11.44

47.11±14.70

41.39±13.30

Male

41.60±16.01

47.65±16.03

47.44±13.32

45.02±13.23

48.98±14.40

45.04±13.84

p value2

0.18

0.00*

0.31

0.15

0.42

0.09

psychology

ro of

caregiver

Jo

ur

na

* Difference significant at p<0.05

re

2. Independent sample t –test

lP

1. One-way analysis of variance (ANOVA)

-p

Gender

23