- Email: [email protected]
Usability Evaluation to Enhance Software Quality of Cultural Conservation System Based on Nielsen Model (WikiBudaya)
Available online at www.sciencedirect.com
ScienceDirect
Available online at www.sciencedirect.com Procedia Computer Science 00 (2018) 000–000
ScienceDirect
www.elsevier.com/locate/procedia
Procedia Computer Science 124 (2017) 513–521
4th Information Systems International Conference 2017, ISICO 2017, 6-8 November 2017, Bali, Indonesia
Usability Evaluation to Enhance Software Quality of Cultural Conservation System Based on Nielsen Model (WikiBudaya) Feby Artwodini Muqtadiroh*, Hanim Maria Astuti, Eko Wahyu Tyas Darmaningrat, Fenty Rizky Aprilian Department of Information Systems, Institut Teknologi Sepuluh Nopember, Surabaya 60111, Indonesia
Abstract WikiBudaya is a web-based encyclopedia application specifically processing information and knowledge on Indonesian culture in forms of articles. It is managed by the Department of Culture and Tourism in East Java Province, Indonesia (DISBUDAPAR JATIM). In the process of its development, the required software quality is not yet controlled. Consequently, WikiBudaya is not yet officially released in accordance with software development life cycle (SDLC). Accordingly, it is necessary to evaluate it by means of software quality control, specifically focused on a usability aspect, i.e.: component of non-functional need to assess the website success. WikiBudaya usability evaluation is based on Nielson Model by adopting user testing method with descriptive statistic data for interpreting the usability quality evaluation. Next, to draw comprehensive results, the result of hypothesis testing is followed up with conclusion of final results in form of recommendations for WikiBudaya website enhancement in order to facilitate better usability by the users and sustainable development. © 2018 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the scientific committee of the 4th Information Systems International Conference 2017. Keywords: website encyclopedia; Indonesian culture; usability evaluation; software quality control; user testing; Nielsen Model; heuristic evaluation
1. Introduction WikiBudaya is a web-based encyclopedia application specifically processing information and knowledge on Indonesian culture in forms of articles and technology repository. This website is designed to bridge the cultural and technological aspects in order to secure that the cultural aspects remain existing in the global era [1]. Unfortunately,
* Corresponding author. Tel.: +62 811 35 22 919; fax: +6231 5964965. E-mail address: [email protected] 1877-0509 © 2018 The Authors. Published by Elsevier B.V.
Peer-review under responsibility of the scientific committee of the 4th Information Systems International Conference 2017.
1877-0509 © 2018 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the scientific committee of the 4th Information Systems International Conference 2017 10.1016/j.procs.2017.12.184
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
514 2
WikiBudaya website is not yet officially released and, so far, the software quality control is not yet implemented. Accordingly, the website quality is not yet well assured. In the process of Software Development Life Cycle [2], WikiBudaya website development is already in testing phase. Accordingly, to assure that it is of high quality, it is to be assessed and controlled to secure its quality assurance before it officially is released [3]. The software quality control applied will focus on the usability aspect, i.e.: nonfunctional need on its usage with the most important and fundamental website parameter as determinant to the success and failure of a website [4]. Under such a circumstance the purpose of this research is to implement software quality control to the WikiBudaya website through testing in form of web usability evaluation to identify its usability based on usability factors suggested in Nielsen Model, namely: learnability, efficiency, memorability, few errors and users’ satisfaction [4]. For this reason, the research proceeds with user testing method where candidate users will have trial on the usage of the web and perceive it through questionnaires. Next, the completed questionnaires are processed and analyzed descriptively and statistically to interpret the results of the usability evaluation. The results of web trial by the users will be validated by means of Heuristic Evaluation technique [5], i.e.: User Interface experts evaluates the web usability in view of User Interface by identifying weaknesses and problems in the web interface and validating the second opinions of the experts to judge the results of web evaluation by the users as well as addressing recommendations for improving the web design of User Interface (UI) based on the evaluation results [6]. Based on the reflections of the web trial, it is expected that the results will be: (1) identification of weaknesses of the web in view of usability aspect, (2) recommendations based on the weaknesses as feedbacks for web usability enhancement for minimizing bugs, improving users’ satisfaction, presenting simple and user-friendly web product designs when officially released. All of these are about the software quality control [7] [3]. 2. Basic Theory 2.1. WikiBudaya WikiBudaya is a web-based encyclopedia application specifically processing information and knowledge on Indonesian culture in forms of articles and technology repository. It is managed by the Department of Culture and Tourism in East Java Province (Disbudapar Jatim) and differently provided with multi-layer of filtrations to validate every article presented by Disbudapar Jatim [1]. 2.2. Software Quality Control Software Quality Control comprises a set of procedures to ensure that the software product satisfied specific attribute purposes in the best point of measure in the view of users by referring to the set functional and nonfunctional requirements [8]. In addition, it also concerns with defect management. 2.3. Nielsen-Model Usability Usability is a quality attribute measuring the extent of simplicity in the interface usages [9]. Nielsen describes schema of receiver success in a system by users and affected by simplicity factors in the following Fig. 1. The measured usability factors comprise learnability, efficiency, memorability, errors and users’ satisfaction.
System Acceptability
Easy to use
Social Acceptability Practical Acceptability
Utility
Easy to learn
Usability
Easy to remember
Usefulness Cost
Few errors Compatibility Reliability Fig. 1. Usability Model by Nielsen
Subjectively pleasing
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
515 3
2.4. User Testing User testing method is an interaction design technique focused on the users for evaluating the software [10] through trials in order to supply with direct inputs based on how users interact with website to accomplish their tasks [11]. This testing measure the product capacity to attain the set goals by measuring the usage simplicity of an object, i.e.: through data capture or collection against task completion time, errors and violations [12]. 3. Conceptual Research Framework The testing is conducted through user testing phases based on Nielsen-Model questionnaires and further evaluated by means of Heuristic Evaluation method, with 3 main phases: Briefing Session, Evaluation Period and Debriefing Session (Fig. 2). The following is the conceptual research framework on evaluation of usability based on Heuristic Evaluation concept [4]:
Fig. 2. Conceptual Research Framework on Evaluation of Usability based on Heuristic Evaluation
With reference to the aforementioned research framework and through the testing on WikiBudaya web usage with real users as the respondents (Briefing Session), it will be further evaluated by adopting Heuristic Evaluation technique for validation or judgment against the results of the website usage trial (Evaluation Period). The judgment by expert evaluator is supposed to address the weaknesses and problems in the WikiBudaya user interface. The outputs from the web usability evaluation will be further adopted for addressing recommendations for the quality of WikiBudaya website usability (Debriefing Session). 4. Research Hypotheses The usability factors in this research become the variables measured. Each variable will be tested and the results of the testing will suggest whether or not each of the variables brings positive effects to the quality of performance of WikiBudaya website usability. This phase is to identify the factors affecting the quality of a website in terms of usability aspects [12]. Learnability (X1) measures the degree of simplicity in executing simple tasks when firstly facing a system design by refereeing to a question: ‘How easily does a user learn to operate the website?’ Efficiency (X2) measures the degree of speed in executing certain tasks after having the web design by refereeing to a question: ‘How quickly are the tasks executed?’ Memorability (X3) measures the degree of speed of a user to recall his skill use a web design when revisiting the web by refereeing to a question: ’How well does a user retain his knowledge after a certain span of time?’ Errors Detected (X4) identifies the number of errors are made by the users, the degree of errors made is and how easy the errors are corrected by refereeing to a question: ‘How many errors are made and what errors are made by the users?’ User Satisfaction (X5) mesaures the degree of satisfaction in using the web design by refereeing to questions: ‘Are users satisfied with the web?’, ‘Do users take large benefits from the system?’, and ‘How long is the system used by the users to help him taking decisions?’ Fig. 3 describes the initial research hypothesis on WikiBudaya website usability measurement. 5. Result and Discussion The first phase is a Pre-User Testing through preparation of a trial scenario. The trial scenario is focused on User Interface (UI) aspect to be operated by users. The scenario consists of the following 5 steps: Searching articles with keywords ‘Tari Pendet and downloading additional files available in articles about ‘Tari Pendent’ (USC01); Searching articles with keyword ‘Denpasar’ and downloading additional files available in articles about ‘Denpasar’ (USC02);
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
516 4
Registering as a contributor and uploading an article required from a contributor (USC03); Logging into WikiBudaya website through available an account and uploading a provided article (USC04); Commenting on 3 articles at random with a legal access right as a contributor (USC05).
Fig. 3. Research Hypotheses on Usability Evaluation
Next is the process of WikiBudaya Website User Testing. The scenario treated before as a test case and executed by the respondents will suggest the executable website functions and the unexecutable ones. The respondent scenario represents as a real user of WikiBudaya website. The total respondents are 24 users. With reference to the user testing results, it is recapitulated as follows that there are three unsuccessful scenarios is described by Table 1. Table 1. Unsuccessful Scenario Nr. Unsuccessful Causes of Failures Scenarios 1 USC02 The system fails to search articles with keywords on place of culture origin 2 USC03 The list function in the system does not work 3
USC05
LAN connection is unstable The navigation of comment function is not clear
Impacts The file is not found and the search is stopped There is no new registered account and the users cannot explore list function The input comments cannot be uploaded It takes quite time to reach for the comment function
In this final phase (post-user testing) the results of website testing will be descriptively and statistically processed in order to identify the tends of respondent behaviors in evaluating the aspects of WikiBudaya website usability. In this phase descriptive statistic testing and inferential testing are executed by SPSS tools. The following are the results of statistical calculations based on indicators in the usability factors as advocated in Nielsen-Model. Mean of Usability The mean describes the degree of acceptance by users as a whole towards the given statement with range 0.00 – 4.00. The means of learnability indicators are described by Table 2 Table 2. The means of learnability indicators Indicator
Evaluation Results
Easy to understand
The respondents agree about the simplicity to understand WikiBudaya website, with a mean of 3.63
Easy to search for specific information
The respondents agree about the simplicity to access certain information in WikiBudaya website, with a mean of 3.70
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
Indicator Easy to identify navigational mechanism
517 5
Evaluation Results The respondents respond neutrally to WikiBudaya website navigation system, with a mean of 3.44
The means of memorability indicators by Table 3. Table 3. The means of memorability indicators Indicator Evaluation Results The respondents agree about the simplicity to memorize WikiBudaya Easy to remember website usage, with a mean of 3.57 Easy to reestablish The respondents respond neutrally to re-access WikiBudaya website with >1 month, with a mean of 3.63
The means of efficiency indicators by Table 4. Table 4. The means of efficiency indicators Indicator Evaluation Results The respondents agree about the simplicity to access information and Easy to reach quickly features and complete their tasks rapidly with WikiBudaya website, with a mean of 3.43 The respondents respond neutrally in regard with easy navigation for their Easy to navigate won or knowledge to WikiBudaya website usage, with a mean of 3.50
The means of error detected indicators by Table 5. Table 5. The means of error detected indicators Indicator Evaluation Results The respondents respond neutrally to the number of errors detected in Few number of errors detected WikiBudaya website usage, with a mean of 3.48 The respondents do not agree about easy error fixing when using Easy to fix WikiBudaya website, with a mean of 2.95.
The means of user satisfaction indicators by Table 6. Table 6. The means of user satisfaction indicators Indicator Evaluation Results The respondents respond neutrally in regard with their perception to System pleasant to use WikiBudaya website system, with a mean of 3.37 The respondents respond neutrally in regard with their comfort in using Comfort to use WikiBudaya website, with a mean of 3.50.
So, mean of (overall) usability indicator: in average the respondents value the usability factor (variable Y) 3.75, meaning that they agree with the given statement. 𝐱𝐱̅ Usability Variable (Y) = 3.75 (Agree) In general, WikiBudaya website meets the users’ expectations to supply them with cultural information and it is useful for specifically searching for cultural information.
5.1. Quality Data Testing A highly valid parameter relatively naturally has only a very small degree of error to assure that the value it indicates is real and true. A validity testing is conducted in this research with product moment correlation, i.e.: by correlating between the scores of given questions and the total score of the variables. The parameter of success in correlation validity testing between the scores of given questions and the total score of the variables refers to the KMO (KaiserMeyer-Oikin) standard.
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
518 6
Here is the range value of Validity Test: 0.81 - 0.9: very good, 0.71 - 0.8: good, 0.61 - 0.7: good enough, 0.5 - 0.6: not too good, > 0.5: invalid. And the value of Cronbach Alpha for Reliability Test must be ≥ 𝟎𝟎. 𝟔𝟔. Table 7 shows a summary of the Quality Data Testing for Usability Evaluation of WikiBudaya. Table 7. Quality Data Testing Result for Usability Evaluation of WikiBudaya Variables Learnability (𝑥𝑥1 ) Memorability (𝑥𝑥2 ) Efficiency (𝑥𝑥3 ) Error Detected (𝑥𝑥4 ) Satisfaction (𝑥𝑥5 )
KMO 0.721 0.614 0.801 0.651 0.739
Validity YES NO
Cronbach Alpha 0,796 0,796 0,826 0,736 0,793
Reliability YES NO
Thus, the results of each item contained five variables in the questionnaire have been declared valid and reliable, so that this research can proceed to be processed in accordance with the purpose of this study, that is measuring the effect of variable X(1,2,3,4,5) to Y(usability). 5.2. Hypotheses Testing There are 3 types of testing to measure the hypotheses whether each hypothesis is accepted or rejected: Normality Test The normality test based on the table below states that the variable of learnability (X1), memorability (X2), Efficiency (X3), Error Detected (X4), Satisfaction (X5), and Usability (Y) have normal distribution. This is represented by the P-value reached over 5%. Table 8 shows the result of normality test on the residual. Table 8. Normality Test Result One-Sample Kolmogorov-Smirnov Test
Unstandardized Residual N
44
Kolmogorov-Smirnov Z
.775
Asymp. Sig. (2-tailed)
.586
Test distribution is Normal.
Multicollinearity Test Between independent variables and other independent variables in the regression is not correlated perfectly or near perfect (independent), it is necessary to establish statistically whether or not multicollinearity symptoms performed by calculating VIF. If VIF is more than 10 means the multicollinearity occurs. The VIF values generated by usability of WikiBudaya variables are shown by Table 9. Table 9. Multicollinearity Test Result Nr. 1 2 3 4 5
Variables Learnability (𝑥𝑥1 ) Memorability (𝑥𝑥2 ) Efficiency (𝑥𝑥3 ) Error (𝑥𝑥4 ) Satisfaction (𝑥𝑥5 )
VIF 2,227 2,127 1,907 3,508 4,231
Result No Multicollinearity No Multicollinearity No Multicollinearity No Multicollinearity No Multicollinearity
Correlation Test Correlation analysis aims to see the closeness of the relationship between two or more variables. The greater the correlation coefficient means the greater the degree of the relationship between two variables. Pearson correlation is usually in a linear relationship (both increased or both decreased). The criteria for correlation coefficient are: very strong > = +/- 0.80, strong = +/- 0.60 - 0.80, medium = +/- 0.40 - 0.60, low = +/- 0.20 - 0.40, and very low <= +/0.20.
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
519 7
This research generated coefficient correlation of usability is shown by Table 10. Table 10. Correlation Test Result Pearson Correlation Value 0,975 0,978 0,974 0,931 0,928
The Effect of X on Y Learnability to the quality of usability Memorability to the quality of usability Efficiency to the quality of usability Errors to the quality of usability User’s satisfaction to the quality of usability
Result Has a very strong correlation Has a very strong correlation Has a very strong correlation Has a very strong correlation Has a very strong correlation
The last is to measure the research model fitness based on the R 2 value. The resulted R2 value is 0.984. It shows that the Learnability (X1), Memorability (X2), Efficiency (X3), Error Detected (X4) and Satisfaction (X5) can confirm that the Usability (Y) is 98.4% and the rest 1,6% is explained by other variables not studied in this research. Based on the results of the above correlation test that is in the range> 0.75 - 1.00 and the model fitness above, then the decision of the hypothesis that has been defined at the beginning of the research decision results are shown by Table 11. Table 11. Hypotheses Result 𝐻𝐻1 =
𝐻𝐻2 = 𝐻𝐻3 = 𝐻𝐻4 = 𝐻𝐻5 =
Hypothesis Learnability Factor (simple to learn) positively affects the WikiBudaya website usability. Memorability Factor (easy to memorize) positively affects the WikiBudaya website usability. Efficiency Factor positively affects the WikiBudaya website usability. Error Factor positively affects the WikiBudaya website usability. User Satisfaction Factor positively affects the WikiBud aya website usability.
Conclusion Accepted Accepted Accepted Accepted Accepted
5.3. Validation of Heuristic Evaluation After passing many kind of testing to get the final result of the research hypothesis, the recommendation to the WikiBudaya to be better are shown by Heuristic Evaluation (HE) techniques then. This technique is supposed to validate the results of website evaluation by the users. Next, the experts will give the judgment to the results of the test done by the users. The following table shows the final results of WikiBudaya usability evaluation based on two distinct perspectives, i.e. : users and expert in user interface. The results This due to the fact that users focus on what they experience and feel when directly using WikiBudaya website, while the experts do on user-interface problems identified and alleged to unfavorably affect the usability. The final result concerns with judgments addressed to evaluate the quality of WikiBudaya usability based on Nielsen Model as presented in the following table. Table 12. Heuristic Evaluation Result on Usability of WikiBudaya Nr.
HEURISTIC PRINCIPLES
FULFILMENT?
USABILITY NIELSEN MODEL
USER PERSPECTIVE
1. 2. 3. 4.
Visibility of the system status Match between system and the real world User control and freedom Consistency and Standards
No Yes No Yes
5
Error Prevention
No
6. 7. 8.
Recognition rather than call Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from errors
Yes No No
System pleasant to use Easy to understand System pleasant to use Easy to re-establish Few numbers of error detected Easy to remember Easy to reach quickly Comfort to use
Yes
Easy to fix
Help and documentation
No
Easy to fix
9. 10.
COMPLIANCE YES
NO
NEUTRAL FULFILLED FULFILLED NEUTRAL
FULFILLED
FULFILLED FULFILLED FULFILLED NOT FULFILLED NOT FULFILLED
520 8
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
So, the following are recommendations of experts in order to enhance the degree of WikiBudaya usability. Providing the content in English language version in order that the Indonesian culture is better exposed. Displaying additional files directly in New Tab when clicked to minimize the users’ efforts and submit them in .pdf format to better secure the validity of the relevant articles. Addressing error messages clearly in order to be easy to understand by the users with highlights to error indications, human readable language usage, soft and without blaming users, problem descriptions, constructive suggestions to cope with problems, clear displays and warning effects. Providing special effects to the web designs by proportionally inserting ornaments, Gif/Flash animations or pictures showing the goals and objectives of the website development, such as : Batik motives, various highlights of popular cultures, or web designs with Indonesian typical themes (Belson & Ho, 2012) Providing a content of number of viewers to reliably develop a trust among the users through indication of accumulated number of viewers. Using maximally 3 kinds of fonts in each web page. The recommended fonts to use are : Arial, Helvetica, Ms Sans Serif Geneva, Lucida Sans Unicode, Lucida Grande, Tahoma, and Geneva in monospace paragraphs. (Belson & Ho, 2012) Preferably using email service in the registration phase. The email is supposed to verify the true identity of each user intending to register himself in WikiBudaya website. (Belson & Ho, 2012) Better considering cross-browser compatibility in the website access. 6. Conclusion The results of evaluation to the model show that learnability, memorability, efficiency, errors, and user’s satisfaction factors have very strong correlation, i.e: positive effects to the quality of WikiBudaya website usability. The performance degrees of WikiBudaya website are as follows: Learnability = 3.60, Memorability = 3.53, Efficiency = 3.46, Error = 3.47, and Users Satisfaction = 3.45. In conclusion, WikiBudaya website usability is only superior in 2 factors: learnability and memorability. This is due to the simplicity of the web functions enabling users easily learn and memorize them. It is perceived to be useful for supplying ample information about Indonesian cultures. Acknowledgements We would like to thank to the Research Center of Institut Teknologi Sepuluh Nopember (LPPM ITS), Surabaya – Indonesia for supporting this research under a grant “Young Researcher Grant” for the year 2016. References [1] A. Muklason, F. A. Muqtadiroh, A. S. Nisafani, and I. Nurkhasanah, “VirtualNUSANTARA: A KNOWLEDGE MANAGEMENT SYSTEM FRAMEWORK FOR CULTURAL HERITAGE AND LOCAL WISDOM CONSERVATION in INDONESIA,” in International Conference in Sustainable Development (ICSD), 2012. [2] R. S. Pressman, Software Engineering A Practitioners’s Approach, Fifth Edit. McGraw-Hill Series in Computer Science, 2001. [3] I. Mistrik, R. M. Soley, J. Grundy, B. Tekinerdogan, and N. Ali, “Defining software quality characteristics to facilitate software quality control and software process improvement,” in Software Quality Assurance, 2016, pp. 47–61. [4] G. . Matera, M; Rizzo, F; Carughi, Web Usability: Principles and Evaluation Methods. 2005. [5] D. Lavery, G. Cockton, and M. Atkinson, “Heuristic Evaluation Usability,” in Evaluation Materials, Department of Computing Science University of Glasgow, 1996, pp. 1-3-5-9. [6] S. Dahal, Eyes Don’t Lie: Understanding First Impression on Website Design using Eye Tracking. Missouri S&T Researchers, 2011. [7] L. Chambers, “Software in Practice IT Quality Control,” 2013. [Online]. Available: http://www.chambers.com.au/glossary/quality_control.php. [8] Fleming, “Clarification of the definitions of SQA and SQC,” 2003. [Online]. Available: http://www.sqa.net/softwarequalitycontrol.html. [9]
G. Nielsen Norman, “Introduction to Usability,” Current http://www.nngroup.com/articles/usability-101-introduction-to-usability/.
Issues
in
Web
Usability,
2014.
[Online].
[10] M. Jackson, S. Crouch, and R. Baxter, “Software Evaluation: Criteria-Based Assessment,” Software Sustainability Institute. 2011. [11] A. Carvalho, USABILITY TESTING OF EDUCATIONAL SOFTWARE: Methods, Technique, and Evaluators. 2009. [12] J. Nielsen, “Usability Engineering,” J. Behav. Inf. Technol., 2004.
Available:
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
[13] IEEE, “730-2014 - IEEE Standard for Software Quality Assurance Processes,” 2014. [14] IEEE, “IEEE Std 1059-1993 IEEE Guide for Software Verification and Validation Plans,” 1993. [15] M. Boulton, “Aesthetic-Usability Effect,” 2005. [Online]. Available: http://www.markboulton.co.uk/journal/aesthetic-usability-effect. [16] H. Belson and J. Ho, “Usability,” Softw. Dev. Tools Technol., pp. 1–28, 2012. [17] H. Belson and J. Ho, Software Development Tools and Technologies. 2010.
521 9
ScienceDirect
Available online at www.sciencedirect.com Procedia Computer Science 00 (2018) 000–000
ScienceDirect
www.elsevier.com/locate/procedia
Procedia Computer Science 124 (2017) 513–521
4th Information Systems International Conference 2017, ISICO 2017, 6-8 November 2017, Bali, Indonesia
Usability Evaluation to Enhance Software Quality of Cultural Conservation System Based on Nielsen Model (WikiBudaya) Feby Artwodini Muqtadiroh*, Hanim Maria Astuti, Eko Wahyu Tyas Darmaningrat, Fenty Rizky Aprilian Department of Information Systems, Institut Teknologi Sepuluh Nopember, Surabaya 60111, Indonesia
Abstract WikiBudaya is a web-based encyclopedia application specifically processing information and knowledge on Indonesian culture in forms of articles. It is managed by the Department of Culture and Tourism in East Java Province, Indonesia (DISBUDAPAR JATIM). In the process of its development, the required software quality is not yet controlled. Consequently, WikiBudaya is not yet officially released in accordance with software development life cycle (SDLC). Accordingly, it is necessary to evaluate it by means of software quality control, specifically focused on a usability aspect, i.e.: component of non-functional need to assess the website success. WikiBudaya usability evaluation is based on Nielson Model by adopting user testing method with descriptive statistic data for interpreting the usability quality evaluation. Next, to draw comprehensive results, the result of hypothesis testing is followed up with conclusion of final results in form of recommendations for WikiBudaya website enhancement in order to facilitate better usability by the users and sustainable development. © 2018 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the scientific committee of the 4th Information Systems International Conference 2017. Keywords: website encyclopedia; Indonesian culture; usability evaluation; software quality control; user testing; Nielsen Model; heuristic evaluation
1. Introduction WikiBudaya is a web-based encyclopedia application specifically processing information and knowledge on Indonesian culture in forms of articles and technology repository. This website is designed to bridge the cultural and technological aspects in order to secure that the cultural aspects remain existing in the global era [1]. Unfortunately,
* Corresponding author. Tel.: +62 811 35 22 919; fax: +6231 5964965. E-mail address: [email protected] 1877-0509 © 2018 The Authors. Published by Elsevier B.V.
Peer-review under responsibility of the scientific committee of the 4th Information Systems International Conference 2017.
1877-0509 © 2018 The Authors. Published by Elsevier B.V. Peer-review under responsibility of the scientific committee of the 4th Information Systems International Conference 2017 10.1016/j.procs.2017.12.184
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
514 2
WikiBudaya website is not yet officially released and, so far, the software quality control is not yet implemented. Accordingly, the website quality is not yet well assured. In the process of Software Development Life Cycle [2], WikiBudaya website development is already in testing phase. Accordingly, to assure that it is of high quality, it is to be assessed and controlled to secure its quality assurance before it officially is released [3]. The software quality control applied will focus on the usability aspect, i.e.: nonfunctional need on its usage with the most important and fundamental website parameter as determinant to the success and failure of a website [4]. Under such a circumstance the purpose of this research is to implement software quality control to the WikiBudaya website through testing in form of web usability evaluation to identify its usability based on usability factors suggested in Nielsen Model, namely: learnability, efficiency, memorability, few errors and users’ satisfaction [4]. For this reason, the research proceeds with user testing method where candidate users will have trial on the usage of the web and perceive it through questionnaires. Next, the completed questionnaires are processed and analyzed descriptively and statistically to interpret the results of the usability evaluation. The results of web trial by the users will be validated by means of Heuristic Evaluation technique [5], i.e.: User Interface experts evaluates the web usability in view of User Interface by identifying weaknesses and problems in the web interface and validating the second opinions of the experts to judge the results of web evaluation by the users as well as addressing recommendations for improving the web design of User Interface (UI) based on the evaluation results [6]. Based on the reflections of the web trial, it is expected that the results will be: (1) identification of weaknesses of the web in view of usability aspect, (2) recommendations based on the weaknesses as feedbacks for web usability enhancement for minimizing bugs, improving users’ satisfaction, presenting simple and user-friendly web product designs when officially released. All of these are about the software quality control [7] [3]. 2. Basic Theory 2.1. WikiBudaya WikiBudaya is a web-based encyclopedia application specifically processing information and knowledge on Indonesian culture in forms of articles and technology repository. It is managed by the Department of Culture and Tourism in East Java Province (Disbudapar Jatim) and differently provided with multi-layer of filtrations to validate every article presented by Disbudapar Jatim [1]. 2.2. Software Quality Control Software Quality Control comprises a set of procedures to ensure that the software product satisfied specific attribute purposes in the best point of measure in the view of users by referring to the set functional and nonfunctional requirements [8]. In addition, it also concerns with defect management. 2.3. Nielsen-Model Usability Usability is a quality attribute measuring the extent of simplicity in the interface usages [9]. Nielsen describes schema of receiver success in a system by users and affected by simplicity factors in the following Fig. 1. The measured usability factors comprise learnability, efficiency, memorability, errors and users’ satisfaction.
System Acceptability
Easy to use
Social Acceptability Practical Acceptability
Utility
Easy to learn
Usability
Easy to remember
Usefulness Cost
Few errors Compatibility Reliability Fig. 1. Usability Model by Nielsen
Subjectively pleasing
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
515 3
2.4. User Testing User testing method is an interaction design technique focused on the users for evaluating the software [10] through trials in order to supply with direct inputs based on how users interact with website to accomplish their tasks [11]. This testing measure the product capacity to attain the set goals by measuring the usage simplicity of an object, i.e.: through data capture or collection against task completion time, errors and violations [12]. 3. Conceptual Research Framework The testing is conducted through user testing phases based on Nielsen-Model questionnaires and further evaluated by means of Heuristic Evaluation method, with 3 main phases: Briefing Session, Evaluation Period and Debriefing Session (Fig. 2). The following is the conceptual research framework on evaluation of usability based on Heuristic Evaluation concept [4]:
Fig. 2. Conceptual Research Framework on Evaluation of Usability based on Heuristic Evaluation
With reference to the aforementioned research framework and through the testing on WikiBudaya web usage with real users as the respondents (Briefing Session), it will be further evaluated by adopting Heuristic Evaluation technique for validation or judgment against the results of the website usage trial (Evaluation Period). The judgment by expert evaluator is supposed to address the weaknesses and problems in the WikiBudaya user interface. The outputs from the web usability evaluation will be further adopted for addressing recommendations for the quality of WikiBudaya website usability (Debriefing Session). 4. Research Hypotheses The usability factors in this research become the variables measured. Each variable will be tested and the results of the testing will suggest whether or not each of the variables brings positive effects to the quality of performance of WikiBudaya website usability. This phase is to identify the factors affecting the quality of a website in terms of usability aspects [12]. Learnability (X1) measures the degree of simplicity in executing simple tasks when firstly facing a system design by refereeing to a question: ‘How easily does a user learn to operate the website?’ Efficiency (X2) measures the degree of speed in executing certain tasks after having the web design by refereeing to a question: ‘How quickly are the tasks executed?’ Memorability (X3) measures the degree of speed of a user to recall his skill use a web design when revisiting the web by refereeing to a question: ’How well does a user retain his knowledge after a certain span of time?’ Errors Detected (X4) identifies the number of errors are made by the users, the degree of errors made is and how easy the errors are corrected by refereeing to a question: ‘How many errors are made and what errors are made by the users?’ User Satisfaction (X5) mesaures the degree of satisfaction in using the web design by refereeing to questions: ‘Are users satisfied with the web?’, ‘Do users take large benefits from the system?’, and ‘How long is the system used by the users to help him taking decisions?’ Fig. 3 describes the initial research hypothesis on WikiBudaya website usability measurement. 5. Result and Discussion The first phase is a Pre-User Testing through preparation of a trial scenario. The trial scenario is focused on User Interface (UI) aspect to be operated by users. The scenario consists of the following 5 steps: Searching articles with keywords ‘Tari Pendet and downloading additional files available in articles about ‘Tari Pendent’ (USC01); Searching articles with keyword ‘Denpasar’ and downloading additional files available in articles about ‘Denpasar’ (USC02);
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
516 4
Registering as a contributor and uploading an article required from a contributor (USC03); Logging into WikiBudaya website through available an account and uploading a provided article (USC04); Commenting on 3 articles at random with a legal access right as a contributor (USC05).
Fig. 3. Research Hypotheses on Usability Evaluation
Next is the process of WikiBudaya Website User Testing. The scenario treated before as a test case and executed by the respondents will suggest the executable website functions and the unexecutable ones. The respondent scenario represents as a real user of WikiBudaya website. The total respondents are 24 users. With reference to the user testing results, it is recapitulated as follows that there are three unsuccessful scenarios is described by Table 1. Table 1. Unsuccessful Scenario Nr. Unsuccessful Causes of Failures Scenarios 1 USC02 The system fails to search articles with keywords on place of culture origin 2 USC03 The list function in the system does not work 3
USC05
LAN connection is unstable The navigation of comment function is not clear
Impacts The file is not found and the search is stopped There is no new registered account and the users cannot explore list function The input comments cannot be uploaded It takes quite time to reach for the comment function
In this final phase (post-user testing) the results of website testing will be descriptively and statistically processed in order to identify the tends of respondent behaviors in evaluating the aspects of WikiBudaya website usability. In this phase descriptive statistic testing and inferential testing are executed by SPSS tools. The following are the results of statistical calculations based on indicators in the usability factors as advocated in Nielsen-Model. Mean of Usability The mean describes the degree of acceptance by users as a whole towards the given statement with range 0.00 – 4.00. The means of learnability indicators are described by Table 2 Table 2. The means of learnability indicators Indicator
Evaluation Results
Easy to understand
The respondents agree about the simplicity to understand WikiBudaya website, with a mean of 3.63
Easy to search for specific information
The respondents agree about the simplicity to access certain information in WikiBudaya website, with a mean of 3.70
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
Indicator Easy to identify navigational mechanism
517 5
Evaluation Results The respondents respond neutrally to WikiBudaya website navigation system, with a mean of 3.44
The means of memorability indicators by Table 3. Table 3. The means of memorability indicators Indicator Evaluation Results The respondents agree about the simplicity to memorize WikiBudaya Easy to remember website usage, with a mean of 3.57 Easy to reestablish The respondents respond neutrally to re-access WikiBudaya website with >1 month, with a mean of 3.63
The means of efficiency indicators by Table 4. Table 4. The means of efficiency indicators Indicator Evaluation Results The respondents agree about the simplicity to access information and Easy to reach quickly features and complete their tasks rapidly with WikiBudaya website, with a mean of 3.43 The respondents respond neutrally in regard with easy navigation for their Easy to navigate won or knowledge to WikiBudaya website usage, with a mean of 3.50
The means of error detected indicators by Table 5. Table 5. The means of error detected indicators Indicator Evaluation Results The respondents respond neutrally to the number of errors detected in Few number of errors detected WikiBudaya website usage, with a mean of 3.48 The respondents do not agree about easy error fixing when using Easy to fix WikiBudaya website, with a mean of 2.95.
The means of user satisfaction indicators by Table 6. Table 6. The means of user satisfaction indicators Indicator Evaluation Results The respondents respond neutrally in regard with their perception to System pleasant to use WikiBudaya website system, with a mean of 3.37 The respondents respond neutrally in regard with their comfort in using Comfort to use WikiBudaya website, with a mean of 3.50.
So, mean of (overall) usability indicator: in average the respondents value the usability factor (variable Y) 3.75, meaning that they agree with the given statement. 𝐱𝐱̅ Usability Variable (Y) = 3.75 (Agree) In general, WikiBudaya website meets the users’ expectations to supply them with cultural information and it is useful for specifically searching for cultural information.
5.1. Quality Data Testing A highly valid parameter relatively naturally has only a very small degree of error to assure that the value it indicates is real and true. A validity testing is conducted in this research with product moment correlation, i.e.: by correlating between the scores of given questions and the total score of the variables. The parameter of success in correlation validity testing between the scores of given questions and the total score of the variables refers to the KMO (KaiserMeyer-Oikin) standard.
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
518 6
Here is the range value of Validity Test: 0.81 - 0.9: very good, 0.71 - 0.8: good, 0.61 - 0.7: good enough, 0.5 - 0.6: not too good, > 0.5: invalid. And the value of Cronbach Alpha for Reliability Test must be ≥ 𝟎𝟎. 𝟔𝟔. Table 7 shows a summary of the Quality Data Testing for Usability Evaluation of WikiBudaya. Table 7. Quality Data Testing Result for Usability Evaluation of WikiBudaya Variables Learnability (𝑥𝑥1 ) Memorability (𝑥𝑥2 ) Efficiency (𝑥𝑥3 ) Error Detected (𝑥𝑥4 ) Satisfaction (𝑥𝑥5 )
KMO 0.721 0.614 0.801 0.651 0.739
Validity YES NO
Cronbach Alpha 0,796 0,796 0,826 0,736 0,793
Reliability YES NO
Thus, the results of each item contained five variables in the questionnaire have been declared valid and reliable, so that this research can proceed to be processed in accordance with the purpose of this study, that is measuring the effect of variable X(1,2,3,4,5) to Y(usability). 5.2. Hypotheses Testing There are 3 types of testing to measure the hypotheses whether each hypothesis is accepted or rejected: Normality Test The normality test based on the table below states that the variable of learnability (X1), memorability (X2), Efficiency (X3), Error Detected (X4), Satisfaction (X5), and Usability (Y) have normal distribution. This is represented by the P-value reached over 5%. Table 8 shows the result of normality test on the residual. Table 8. Normality Test Result One-Sample Kolmogorov-Smirnov Test
Unstandardized Residual N
44
Kolmogorov-Smirnov Z
.775
Asymp. Sig. (2-tailed)
.586
Test distribution is Normal.
Multicollinearity Test Between independent variables and other independent variables in the regression is not correlated perfectly or near perfect (independent), it is necessary to establish statistically whether or not multicollinearity symptoms performed by calculating VIF. If VIF is more than 10 means the multicollinearity occurs. The VIF values generated by usability of WikiBudaya variables are shown by Table 9. Table 9. Multicollinearity Test Result Nr. 1 2 3 4 5
Variables Learnability (𝑥𝑥1 ) Memorability (𝑥𝑥2 ) Efficiency (𝑥𝑥3 ) Error (𝑥𝑥4 ) Satisfaction (𝑥𝑥5 )
VIF 2,227 2,127 1,907 3,508 4,231
Result No Multicollinearity No Multicollinearity No Multicollinearity No Multicollinearity No Multicollinearity
Correlation Test Correlation analysis aims to see the closeness of the relationship between two or more variables. The greater the correlation coefficient means the greater the degree of the relationship between two variables. Pearson correlation is usually in a linear relationship (both increased or both decreased). The criteria for correlation coefficient are: very strong > = +/- 0.80, strong = +/- 0.60 - 0.80, medium = +/- 0.40 - 0.60, low = +/- 0.20 - 0.40, and very low <= +/0.20.
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
519 7
This research generated coefficient correlation of usability is shown by Table 10. Table 10. Correlation Test Result Pearson Correlation Value 0,975 0,978 0,974 0,931 0,928
The Effect of X on Y Learnability to the quality of usability Memorability to the quality of usability Efficiency to the quality of usability Errors to the quality of usability User’s satisfaction to the quality of usability
Result Has a very strong correlation Has a very strong correlation Has a very strong correlation Has a very strong correlation Has a very strong correlation
The last is to measure the research model fitness based on the R 2 value. The resulted R2 value is 0.984. It shows that the Learnability (X1), Memorability (X2), Efficiency (X3), Error Detected (X4) and Satisfaction (X5) can confirm that the Usability (Y) is 98.4% and the rest 1,6% is explained by other variables not studied in this research. Based on the results of the above correlation test that is in the range> 0.75 - 1.00 and the model fitness above, then the decision of the hypothesis that has been defined at the beginning of the research decision results are shown by Table 11. Table 11. Hypotheses Result 𝐻𝐻1 =
𝐻𝐻2 = 𝐻𝐻3 = 𝐻𝐻4 = 𝐻𝐻5 =
Hypothesis Learnability Factor (simple to learn) positively affects the WikiBudaya website usability. Memorability Factor (easy to memorize) positively affects the WikiBudaya website usability. Efficiency Factor positively affects the WikiBudaya website usability. Error Factor positively affects the WikiBudaya website usability. User Satisfaction Factor positively affects the WikiBud aya website usability.
Conclusion Accepted Accepted Accepted Accepted Accepted
5.3. Validation of Heuristic Evaluation After passing many kind of testing to get the final result of the research hypothesis, the recommendation to the WikiBudaya to be better are shown by Heuristic Evaluation (HE) techniques then. This technique is supposed to validate the results of website evaluation by the users. Next, the experts will give the judgment to the results of the test done by the users. The following table shows the final results of WikiBudaya usability evaluation based on two distinct perspectives, i.e. : users and expert in user interface. The results This due to the fact that users focus on what they experience and feel when directly using WikiBudaya website, while the experts do on user-interface problems identified and alleged to unfavorably affect the usability. The final result concerns with judgments addressed to evaluate the quality of WikiBudaya usability based on Nielsen Model as presented in the following table. Table 12. Heuristic Evaluation Result on Usability of WikiBudaya Nr.
HEURISTIC PRINCIPLES
FULFILMENT?
USABILITY NIELSEN MODEL
USER PERSPECTIVE
1. 2. 3. 4.
Visibility of the system status Match between system and the real world User control and freedom Consistency and Standards
No Yes No Yes
5
Error Prevention
No
6. 7. 8.
Recognition rather than call Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from errors
Yes No No
System pleasant to use Easy to understand System pleasant to use Easy to re-establish Few numbers of error detected Easy to remember Easy to reach quickly Comfort to use
Yes
Easy to fix
Help and documentation
No
Easy to fix
9. 10.
COMPLIANCE YES
NO
NEUTRAL FULFILLED FULFILLED NEUTRAL
FULFILLED
FULFILLED FULFILLED FULFILLED NOT FULFILLED NOT FULFILLED
520 8
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
So, the following are recommendations of experts in order to enhance the degree of WikiBudaya usability. Providing the content in English language version in order that the Indonesian culture is better exposed. Displaying additional files directly in New Tab when clicked to minimize the users’ efforts and submit them in .pdf format to better secure the validity of the relevant articles. Addressing error messages clearly in order to be easy to understand by the users with highlights to error indications, human readable language usage, soft and without blaming users, problem descriptions, constructive suggestions to cope with problems, clear displays and warning effects. Providing special effects to the web designs by proportionally inserting ornaments, Gif/Flash animations or pictures showing the goals and objectives of the website development, such as : Batik motives, various highlights of popular cultures, or web designs with Indonesian typical themes (Belson & Ho, 2012) Providing a content of number of viewers to reliably develop a trust among the users through indication of accumulated number of viewers. Using maximally 3 kinds of fonts in each web page. The recommended fonts to use are : Arial, Helvetica, Ms Sans Serif Geneva, Lucida Sans Unicode, Lucida Grande, Tahoma, and Geneva in monospace paragraphs. (Belson & Ho, 2012) Preferably using email service in the registration phase. The email is supposed to verify the true identity of each user intending to register himself in WikiBudaya website. (Belson & Ho, 2012) Better considering cross-browser compatibility in the website access. 6. Conclusion The results of evaluation to the model show that learnability, memorability, efficiency, errors, and user’s satisfaction factors have very strong correlation, i.e: positive effects to the quality of WikiBudaya website usability. The performance degrees of WikiBudaya website are as follows: Learnability = 3.60, Memorability = 3.53, Efficiency = 3.46, Error = 3.47, and Users Satisfaction = 3.45. In conclusion, WikiBudaya website usability is only superior in 2 factors: learnability and memorability. This is due to the simplicity of the web functions enabling users easily learn and memorize them. It is perceived to be useful for supplying ample information about Indonesian cultures. Acknowledgements We would like to thank to the Research Center of Institut Teknologi Sepuluh Nopember (LPPM ITS), Surabaya – Indonesia for supporting this research under a grant “Young Researcher Grant” for the year 2016. References [1] A. Muklason, F. A. Muqtadiroh, A. S. Nisafani, and I. Nurkhasanah, “VirtualNUSANTARA: A KNOWLEDGE MANAGEMENT SYSTEM FRAMEWORK FOR CULTURAL HERITAGE AND LOCAL WISDOM CONSERVATION in INDONESIA,” in International Conference in Sustainable Development (ICSD), 2012. [2] R. S. Pressman, Software Engineering A Practitioners’s Approach, Fifth Edit. McGraw-Hill Series in Computer Science, 2001. [3] I. Mistrik, R. M. Soley, J. Grundy, B. Tekinerdogan, and N. Ali, “Defining software quality characteristics to facilitate software quality control and software process improvement,” in Software Quality Assurance, 2016, pp. 47–61. [4] G. . Matera, M; Rizzo, F; Carughi, Web Usability: Principles and Evaluation Methods. 2005. [5] D. Lavery, G. Cockton, and M. Atkinson, “Heuristic Evaluation Usability,” in Evaluation Materials, Department of Computing Science University of Glasgow, 1996, pp. 1-3-5-9. [6] S. Dahal, Eyes Don’t Lie: Understanding First Impression on Website Design using Eye Tracking. Missouri S&T Researchers, 2011. [7] L. Chambers, “Software in Practice IT Quality Control,” 2013. [Online]. Available: http://www.chambers.com.au/glossary/quality_control.php. [8] Fleming, “Clarification of the definitions of SQA and SQC,” 2003. [Online]. Available: http://www.sqa.net/softwarequalitycontrol.html. [9]
G. Nielsen Norman, “Introduction to Usability,” Current http://www.nngroup.com/articles/usability-101-introduction-to-usability/.
Issues
in
Web
Usability,
2014.
[Online].
[10] M. Jackson, S. Crouch, and R. Baxter, “Software Evaluation: Criteria-Based Assessment,” Software Sustainability Institute. 2011. [11] A. Carvalho, USABILITY TESTING OF EDUCATIONAL SOFTWARE: Methods, Technique, and Evaluators. 2009. [12] J. Nielsen, “Usability Engineering,” J. Behav. Inf. Technol., 2004.
Available:
Feby Artwodini Muqtadiroh et al. / Procedia Computer Science 124 (2017) 513–521 Feby Artwodini Muqtadiroh, et al./ Procedia Computer Science 00 (2018) 000–000
[13] IEEE, “730-2014 - IEEE Standard for Software Quality Assurance Processes,” 2014. [14] IEEE, “IEEE Std 1059-1993 IEEE Guide for Software Verification and Validation Plans,” 1993. [15] M. Boulton, “Aesthetic-Usability Effect,” 2005. [Online]. Available: http://www.markboulton.co.uk/journal/aesthetic-usability-effect. [16] H. Belson and J. Ho, “Usability,” Softw. Dev. Tools Technol., pp. 1–28, 2012. [17] H. Belson and J. Ho, Software Development Tools and Technologies. 2010.
521 9