- Email: [email protected]
Data Mining Oriented Software Quality Estimation
Available online at www.sciencedirect.com
ScienceDirect Procedia Computer Science 91 (2016) 1028 – 1037
Information Technology and Quantitative Management (ITQM 2016)
Data Mining oriented Software Quality Estimation Shusaku Tsumotoa,1 , Shoji Hiranoa a Department
of Medical Informatics, Faculty of Medicine, Shimane University, 89-1 Enya-cho, Izumo 693-8501 Japan
Abstract Clinical environment is very complex, and flexible and adaptive service improvement is crucial in maintaining quality of medical care. Thus, incremental update of software service in a hospital information system (HIS) and its evaluation is important. This paper introduces an active mining process for development of a an embedded software in which service logs stored in HIS are used to calculate the test statistics for evaluation on the effect of introduction of a new alarming service for clinical practice. are used to measure the differences The results show that proposed method is useful to evaluate the system performance in a real clinical environment. c 2016 2016Published The Authors. Published Elsevier B.V. © by Elsevier B.V. by This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Selection and/or peer-review under responsibility of ITQM2016. Peer-review under responsibility of the Organizing Committee of ITQM 2016
Keywords: software quality estimation process; active mining; hospital information system
1. Introduction Hospital information system (HIS)[1] has been introduced in a large-scale hospital and computerized clinical process. Stored data include all the results of clinical actions in a hospital, including laboratory examinations, physical examinations and clinical decisions. Since all the stored information are easily searched and visualized, the quality of hospital service depends on the quality of HIS. HIS improves the traceability of medical services: all the clinical actions are recorded with issued and executed time. Secondly, the overvall computerized service supports not only medical decisions and hospital management by using the stored data. However, these subjects have been rarely examined in a quantitative way. With respect to the first improvement, Hbner-Bloder and Ammenwerth[2] use performance indicators calculated from a questionnaire to the patients and doctors and the changes are detected before and after HIS has been introduced, although they can only evaluate overall performance, which cannot detect the details of improvements. Concerning the latter observation, Anema et al[3] use the clinical indictors for the quality of surgical operations. Although these studies give considerable insights, the analysis is not based on data stored in HIS. Since HIS keeps the records of clinical actions, evaluation can be made by using stored data. Although Tsumoto et al. [4, 5] applied to data mining methods for this purpose, the results are very naive. Another important point is that HIS is not complete. Since clinical environments are complex, all the requirements cannot be retrieved from the clinical scend, and developement of HIS should be incremental. However, the effect of improvements is not so easy to evaluate and subjective studies have been mainly used for evaluation. ∗ Corresponding author. Tel.: +81-853-20-2171 ; fax: +81-853-20-2170. E-mail address: [email protected].
1877-0509 © 2016 Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Organizing Committee of ITQM 2016 doi:10.1016/j.procs.2016.07.141
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
This paper proposes an active-mining framework for evaluation of improved software as follows. First, a problem pointed out by medical staff is selected and related data is extracted from HIS. Second, applying data mining methods, the characteristics of the problem are analyzed. Then, medical staff interprets the results and discuss the solutions. Proposed solutions are implemented on a hospital information system and the system keeps tracks of the usage of the improved program. Then, finally, the service is evaluated by the analysis of stored service logs. We focused on one practical problem for evaluation of the problem in our university hospital illustrated the effectiveness of this framework, based on active mining cycle[6]. 2. Motivation 2.1. Architecture of HIS HIS [1] follows service-oriented architecture and keeps records of all the clinical actions, including laboratory examinations, physical examinations, medical decision making, nursing actions, and so on. Incident or accident reports are not exception: they are also stored in HIS as clinical databases. The important characteristics of HIS is that the stored data includes spatial and temporal information: when are where orders are issued and executed. For example, Figure 1 shows the architecture of the HIS in Shimane University Hospital. HIS can be viewed Infection Control Support ing System
ECG Filing System Radiogram Reporting System
Electrophysiologocial Examination Supporting System
Respiratory Function Filing System
Interface to Filming Equipments
Ultrasonic Filing System
Reference Image/Interpretation PACS
Blood Collection Prepartion
Pathological Examinations Reporting System
Clinical Laboratory Examination Managment
Clinical Trial Management system
Data Mining System
Radiology Information System
Pharmaceutical Dept Managment
Bacterial Test
Rehabilitation
Radiology Department
Regional Heathcare Supporting System
Blood Transfusion Management Syssten
Narcotic Drug Management
Endoscopic Image Filing System
Operation Department
Blood Sample
Blood Transfusion Department
Laboratory Examination Department
Drug administration Guidance Endoscope Dept
Pathology Department
Rehabilitation Department
Pharmaceutical Department
Analysis for Managment
Medicines Stock Management System
Medical Informatics
Hospital Information System
Logistics Supporting System Incident Reporting System
Nutritional Management Department
㪤㪼㪻㫀㪺㪸㫃㩷㪠㫅㪽㫆㫉㫄㪸㫋㫀㪺㫊㩷 Division of Medical Informatics Order-Entry System Hospitalize/Release Food Service Reservation
Physiological Exam Blood Transfusion Radiology Labo Exam
Prescription Injection
Rehabilitation Medical Procedure
Pathology
DPC
Disease Type
Operation
Electronic Patient Records
Medical Professions Division
Food Service Supporting System
Nursing Support System Medical Service Payment System
Patient Records Authorization EPR viwer Patient Selection Drawing Memo
Patient Registration Card Bedside Entry Medical Document
Outpatient Clinic Management
Scanning System Discharge Summary
Datawarehouse for Medical Business
Fig. 1. Hospital Information System
as a large-scale network, through wich all the clinical information are exchanged as a cyberspace embedded in a hospital, keeping track of temporal information. By using service log, medical service can be visualized. Tsumoto et al. investigates the principal factor related with length of stay and medical payment by applying univariate and multivariate statistical analysis to data of Chiba Universiy Hospital [7, 8]. However, the most important part of stored informartion is “temporal”. Focusing on this aspects, Tsumoto et al. [9, 10] applied temporal data mining methods to capture the characteristics of the university hospital. 2.2. Order in HIS The basic information unit of HIS is an “order”, since HIS was firstly introduced as a order-entry system. For example, prescription is an order from a doctor to a pharmacist who will prepare for the pills for the therapy. Workflow of a prescription order will be shown as:
1029
1030
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
1. 2. 3. 4. 5. 6. 7.
Outpatient Clinic A prescription given from a doctor to a patient The patient bring it to medical payment department The patient bring it to pharmaceutical department Execution of order in pharmacist office Delivery of prescribed medication Payment
Figure 2 illustrates the information flow of prescription order, which can be implemented by UML. The second to fourth steps can be viewed as information propagation through the network. HIS keeps all the executed process of the prescription based on the information flow shown in Figure 2 as service logs.
Clinic
Pharmaceutical Department Prescription (Issued Order)
Pills Execution
Patient
Bill Order Medical Payment
Fig. 2. Workflow of Prescription Order
2.3. Patient Flow at Outpatient Clinic Figure 3 below shows a patient’s direction in a hospital for an outpatient clinic visit in Japan, which can be partially traced by the log of the hospital information system. First, a patient registers for their appointment at the reception desk. Then, the patient moves to a laboratory examination room and pre-ordered laboratory tests are performed. Next, the patient arrives at an examination room and waits for a physical examination by the doctor. During this process, the doctor will issue orders, such as prescriptions, further laboratory tests and other examinations, and reserve a time for the patient for his/her next visit. Finally, the patient will go back to the reception again to pay for services rendered, and the patient will be given a receipt and a paper for his/her future reservation. Figure 3 below illustrates patients movement in outpatient clinic. A patient first visits the reception desk, and then goes to laboratory examination room. Then, after laboratory examinations, he/she goes to the outpatient clinic and waits for calling from a doctor. When a doctor calls a patient, the doctor interviews the patient, starts physical examinations and evaluate all the examinations. Finally, the doctor issues prescription and reserve the next visit with reservation of laboratory examination.
Reception
Reception
Execution of Order (Laboratory Examination)
Issuance of Order Clinic (Physical Examination)
Label Printing Arrival at an examination room
(Prescription, Examinations, Reservation to Next Visit)
Start of Examination
Reception (Payment,Handout of Paper for Reservation)
End of Examination
Reserved Time
Fig. 3. Patient Flow in Outpatient Clinic
3. Method 3.1. Active Mining Process Motoda et al [11] proposed activing mining process in 2002, which is a comprehensive data mining and decision making process, as summarized in Figure 4. In the process, first we apply data mining methods to the
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
original data (active-user oriented mining). Then, the user evaluate the mining results, which may lead to a new discovery (active user reaction). Next, according to the users intepretation, new information gathering will be performed and the new stage for data collection (active information gathering) will be evoked. After enough data are stored, data mining process will be applied again as a active mining cycle.
3.2. Software Quality Estimation Process Based on the active mining process, we propose the fo f llowing software f development process as shown in Figure 5. First, data extracted from hospital information system is used to characterize a given problem where software development is important for a solution (information gathering). After temporal data mining methods are applied (data mining), the results are interpreted by medical staff and the solutions are discussed (active user reaction). Based on the discussions, new programs will be developed and service logs will be stored for its execution (active informat f ion gathering). Then, their performance f was evaluated using the service logs by using data mining methods (data mining). Data mining with Service Logs
Hypothesis Generation
Data Integration
Hypothesis V rification Ve
Software Implementation
Service Execution (Storage of New Logs)
Data in HIS Fig. 5. Software Quality Estimation Process
3.3. Experimental Setting and Quality t Evaluation In this paper, the data used for hypothesis generation is the chronological change of the number of clinical orders and waiting time. Analytical method of temporal data is based on multiscale matching method proposed in [9]. Concerning the meaning of chronological change of the number of orders and the analysis method, the reader could refer to [10].
1031
1032
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Experiments were conducted from fiscal year 2010 to 2012. In the ends of fiscal year 2010 and 2011, the new softwares were embedded into hospital information system and service logs are collected. Service logs which records the patient movement were extracted, and the time difference ff between visit to each division was calculated. Fiscal year 2010 gives the baseline, the first and second improvement was implemented in 2011 and 2012, repectively. Statistical analyses, including Kruskal-Wallis test were applied to statistics obtained from service logs, where R3-1-4 was used. 4. Experimental Results 4.1. First Cycle: Hypothesis Generation (Data Mining) Figure 6 shows a temporal chart of the number of orders on Tuesday in fiscal 2010. Vertical and horizonal axes show the number of orders and each timing slot, whose unit is one hour. The peak of each order are almost located between 9am to 5pm, corresponding to opening hours of outpatient clinic. The shape of the temporal curve characterize each clinical division, which can be captured by temporal data mining methods where temporal data mining methods, as discussed in [10]. For example, Figure 7 and 8 shows the temporal charts of hepatology
Fig. 6. Temporal Chart of Total Number of Orders (Fiscal Year 2010):Vertical axis and horizontal axis denote the service hours from 0 to 23 hours and the number of orders. Each point plots the number of orders during the time interval. For example, points on 9 of the vertical axis show the number of orders counted from 9:00 to 9:59.
Fig. 7. Temporal Chart of Number of Orders (Hepatology, Fiscal Year 2010)
and rheumatology in 2010. While the patterns of Figure 7 is very similar to Figure 6, Figure 8 is different ff from
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Prescription Examination Radiology Operation Transfusion Meal Pathology Injections Reservation Document Nursing Procedure Record Rehabilitation Admission
Fig. 8. Temporal Chart of Number of Orders (Rheumatology, Fiscal Year 2010)
those as follows. First, the peak of laboratory examination of rheumatology is in 9 to 10am, though those of others are located around 12pm. Secondly, records and reservation have the high numbers from 10am to 5pm, compared with other orders. Thus, the obtained results shows that a rhuematologist use a HIS interface in a different ff way from a hepatoligst whose workflow in the outpatient clinic reflects the ordinary pattern. This hypothesis will not only lead to the temporal change of the number orders, but also influence over service execution time for a rheumatologist. So, the next step is to measure the effect ff of the workflow change, which may be captured by temporal interval between these steps: the time difference ff between reception and execution of order, that between execution of order and its report, that between reporting and physical examination. Also, since each patient has a reserved timeslot, the temporal difference ff between reservation and actual examination is important, because this factor is related with the patients’ satisfaction. Table 1 shows the statistics of two divisions in 2010. The first column shows that ten minutes and 15 minutes are needed for laboratory examinations in the case of hepatology and rheumatology. It is notable that the medians are equal, which suggests that a significant number of patients of rheumatology waits longer than ones of hepatology. Table 1. Statistics of Waiting Time in 2010
Reception-Lab Lab-Lab Report LaboReport-Clinic Reservation-Clinic
Hepatology Average Median 10.48 4 25.22 20.0 96.83 77.0 48.45 31.0
Rhematology Average Median 14.83 4 19.0 17 95.58 77.0 51.82 48.0
4.2. First Cycle: Hypothesis Verification and Software Development The obtained results were shown to staff of laboratory and rheumatology, the following two points were found for Figure 8. First, during the peak of laboratory examinations, laboratory staff found missing orders and asked the doctors to reissue the orders. Then, secondly, although rheumatologists usually issued orders after their clinic has been finished, they sometimes forgot to book some orders because they left many orders not issued. Usually, the doctors should issue orders during physical examinations, but rheumatologists postpone booking due to their time constratins. The statistics shows they examined about 6 to 7 patients per hour, totally 30-35 patients on Tuesday. In the avarage, seven of them have missing orders in total. So the rate for forgetting orders can be estimated as 20-23%, This is the main reason why the patients wait for longer. Thus, the second discovery is the cause of the first discovery, which should be improved as a first step.
1033
1034
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Table 2. Statistics of Waiting Time of Hepatology from 2010 to 2013
(min)
Reception to Laboratory
Laborary Examination
2010 2011 2012 2013
10.17 ± 19.31 10.48 ± 20.00 8.94 ± 22.38 7.79 ± 20.29
23.95 ± 17.53 25.21 ± 17.25 27.23 ± 14.98 26.04 ± 13.60
Reporting to Inspection Reporting to Inspection 85.60 ± 66.19 96.83 ± 79.18 92.14 ± 73.68 95.57 ± 73.85
Reservation to Inspection 36.16 ± 58.00 48.45 ± 76.20 44.10 ± 67.37 43.94 ± 65.67
Figure 9 shows the workflows of outpatient clinic of hepatology and rheumatology after the discussion, whose manifestations are shown in Figure 7 and 8 , respectively. Thus, if the forget errors are improved, the temporal chart of laboratory examinations of rheumatology may be near to that of hepatology. Also, the changes can be measured as improvements of waiting time.
Hepatology Reception
Execution of Order (Laboratory Examination)
Clinic
Issuance of Order
Execution of Order
Clinic
Issuance of Order?
Rheumatology Reception
Issuance of Order Fig. 9. Workflow of Outpatient Clinic of Hepatology and Rheumatology Confirmed by Interview with Doctors
In fiscal year 2011, the alarm was implemented for the solution of the second problem: when a doctor wrote down the comment in the reservation confirmation, such as laboratory examinations needed, but left the orders not issued during the visit and planned to issue after all the clinic had been finished. However, the doctors forgot to issue them. Then, during the next visit the missing orders were found at laboratory examination room, and the laboratory staff would call the doctors and ask them to issue an order. To improve the situation, the first step was to introduce a alerming system for the discrepancy between reservation confirmation and issued order: if it detected before they closed a window for the patient, alerm will be evoked. 4.3. Second Cycle One year trial showed that waiting time has a small improvement. So, we interviewed the rheumatologist and prepared for the next solution: implementation of management of the orders when they finished their clinic and before they logged off in fiscal year 2012: When the doctor is going to log off, the management screen of orders were poped up to check all the forgot orders where the doctors issue orders from that screen. 4.4. Evaluation Results of First and Second Cycles 4.4.1. Change from 2010 to 2013 (Hepatology) Figure 10 shows the change of waiting times from 2010 to 2013 in Hepatology. cTable 3 shows the results of Kruskal-Wallis in terms of the time interval between reception and execution of laboratory examinations, which reflects the results obtained from Figure 10 and Table 2.
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037 Averaged Time needed between Laboratory Exam and Reporting
26 Time 24
8
25
9
Waiting Time
10
27
11
Averaged Waiting Time between Reception and Laboratory Exam
n=4246
n=4148
n=4212
n=4013
n=4246
n=4148
n=4212
n=4013
2010fy
2011fy
2012fy
2013fy
2010fy
2011fy
2012fy
2013fy
Fiscal Year
Averaged Time needed between Laboratory Reporting and Examination by the Doctors
Averaged Waiting Time between Reserved Time and Examination by the Doctors
Time 35
85
40
90
Time
45
95
50
Fiscal Year
n=4246
n=4148
n=4212
n=4013
n=4246
n=4148
n=4212
n=4013
2010fy
2011fy
2012fy
2013fy
2010fy
2011fy
2012fy
2013fy
Fiscal Year
Fiscal Year
Fig. 10. Change of Hepatology in 2010 to 2013. Vertical and horizontal axes denote the fiscal year and time needed for executing services.
Table 3. Kruskal-Wallis Test (Reception and Execution of Laboratory Examinations (p-value))
2010 2011 2012
2010 −
2011 0.900 −
2012 0.031 0.003 −
2013 < 0.001 < 0.001 0.052
4.4.2. Change from 2010 to 2013 (Rheumatology) Figure 11 plots the transition from 2010 to 2013 in Rhuematology. Since the values are monotonically improving, both of the interface implementation was effective. ff Table 5 shows the results of Kruskal-Wallis test in terms of to the time interval between reception and execution of laboratory examinations. However, the difference ff between 2010 and 2011 is not statistically significant. The differences ff also show that the second improvement is much better than the first one also for rheumatology (Table 4). However, the other waiting times increased, so total waiting time were not changed. Thus, although patient satisfaction may have been improved for laboratory examination, overall satisfaction may not. 5. Discussion Although quantative evaluation of HIS is important with respect to improvements of medical service quality, empirical evaluation has just started as shown in [12]. The difficulty is due to hardness of automated data acquisition for evaluation. Usually, questionnaire of end-users are used for estimation of the system, which is frequently used for software evaluation in other fields. However, it should be pointed software services are now embedded into large-scale software and that histories of execution are automatically accumulated, which can be viewed as “big data”. Thus, the way how to use the stored big data is indispensable for the improvements in embedded
1035
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Averaged Time needed between Laboratory Exam and Reporting
Time
20.5
13 10
19.5
11
20.0
12
Waiting Time
14
21.5
15
22.0
16
22.5
Averaged Waiting Time between Reception and Laboratory Exam
21.0
1036
n=2660
n=2955
n=3880
n=3680
2010fy
2011fy
2012fy
2013fy
n=2660
n=2955
n=3880
n=3680
2010fy
2011fy
2012fy
2013fy
Fiscal Year
Averaged Time needed between Laboratory Reporting and Examination by the Doctors
Averaged Waiting Time between Reserved Time and Examination by the Doctors
50
95
55
60
Time
105 100
Time
110
65
115
70
Fiscal Year
n=2660
n=2955
n=3880
n=3680
n=2660
n=2955
n=3880
n=3680
2010fy
2011fy
2012fy
2013fy
2010fy
2011fy
2012fy
2013fy
Fiscal Year
Fiscal Year
Fig. 11. Change of Rheumatology in 2010 to 2013. Vertical and horizontal axes denote the fiscal year and time needed for executing services.
Table 4. Statistics of Waiting Time of Rheumatology from 2010 to 2013
(min)
Reception to Laboratory
2010 2011 2012 2013
14.83 ± 32.50 13.64 ± 29.64 11.16 ± 30.52 10.85 ± 28.52
Laborary Examination to Reporting 19.98 ± 13.67 21.34 ± 13.33 22.05 ± 11.08 22.01 ± 11.12
Reporting to Inspection
Reservation to Inspection
95.58 ± 84.60 96.16 ± 81.6 115.71 ± 101.70 101.83 ± 85.07
51.82 ± 46.21 51.08 ± 47.88 69.05 ± 76.70 54.68 ± 47.93
services. But, it should be pointed out that such data storage and analyis are based on “retrospective” platform, whose evidence is weaker than experimental designed scheme, called “prospective analysis”. Thus, “big data” analytics should be curated by such a data collection form. Thus, active mining process may give a nice platform prospective analysis because it can consider the effects ff of intervention in a comprehensive way. The experiments and their results shown in this paper can be viewed as the first achievement of active-mining oriented software evaluation process. 6. Conclusion This paper proposes the following active-mining centered software development process. First, a problem pointed out by medical staff is selected and related data is extracted from HIS. Second, applying data mining methods, the characteristics of the problem are analyzed. Then, the results obtained are interpreted by medical staff and the solutions will be discussed. Proposed solutions are implemented on a hospital information system and the system keeps tracks of the usage of the improved program. Then, finally, the service is evaluated by the analysis of stored service logs.
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Table 5. Kruskal-Wallis Test (Reception and Execution of Laboratory Examinations (p-value))
2010 2011 2012
2010 −
2011 0.452 −
2012 < 0.001 0.0042 −
2013 < 0.001 0.0011 0.964
The process was evaluated in Shimane Unversity Hospital. Two divisions, hepatology and rheumatology were selected for comparison. The obtained mining results gave a hypothesis that the workflow of rheumatology is different from that of hepatology, which reflected the ordinary workflow in the outpatient clinics of the university hospital and caused the problem where experts forgot to issue the orders. The first step for solution was to implement an alarming: if a doctor input a comment on laboratory examination in a reservation sheet but has not yet issued an order before closing the file, an alert will be evoked. However, after one year trial, statistics showed a small improvement in waiting time: after the discussion, in fiscal year 2012, another alarming screen was implemented where all the forgotten orders for patients who visited that day would be displayed, and the doctors could go back to issue orders. The workflow and waiting time were improved after the second installation. The process, which can be viewed as application of active mining process, will give a new framework for quantitative evaluation of software development in hospital information system. Acknowledgements This research is supported by Grant-in-Aid for Scientific Research (B) 15H2750 from Japan Society for the Promotion of Science(JSPS). References [1] E. Shortliffe, J. Cimino (Eds.), Biomedical Informatics: Computer Applications in Health Care and Biomedicine, 3rd Edition, Springer, 2006. [2] G. Hubner-Bloder, E. Ammenwerth, Key performance indicators to benchmark hospital information systems - a delphi study, Methods Inf Med (6) (2009) 508518. [3] H. Anema, J. Kievit, C. Fischer, E. Steyerberg, N. Klazinga, Inuences of hospital information systems, indicator data collection and computation on reported dutch hospital performance indicator scores, BMC Health Serv Res. (2013) 212. [4] S. Tsumoto, Knowledge discovery in clinical databases and evaluation of discovered knowledge in outpatient clinic, Information Sciences (124) (2000) 125–137. [5] S. Tsumoto, G5: Data mining in medicine, in: W. Kloesgen, J. Zytkow (Eds.), Handbook of Data Mining and Knowledge Discovery, Oxford University Press, Oxford, 2001, pp. 798–807. [6] S. Tsumoto, T. Yamaguchi, M. Numao, H. Motoda (Eds.), Active Mining, Second International Workshop, AM 2003, Maebashi, Japan, October 28, 2003, Revised Selected Papers, Vol. 3430 of Lecture Notes in Computer Science, Springer, 2005. [7] Y. Tsumoto, S. Tsumoto, Exploratory univariate analysis on the characterization of a university hospital: A preliminary step to datamining-based hospital management using an exploratory univariate analysis of a university hospital, The Review of Socionetwork Strategies 4 (2) (2010) 47–63. [8] Y. Tsumoto, S. Tsumoto, Correlation and regression analysis for characterization of university hospital (submitted), The Review of Socionetwork Strategies 5 (2) (2011) 43–55. [9] S. Tsumoto, S. Hirano, Detection of risk factors using trajectory mining, J. Intell. Inf. Syst. 36 (3) (2011) 403–425. doi:10.1007/s10844009-0114-7. [10] S. Tsumoto, H. Iwata, S. Hirano, Y. Tsumoto, Similarity-based behavior and process mining of medical practices, Future Generation Comp. Syst. 33 (2014) 21–31. doi:10.1016/j.future.2013.10.014. [11] H. Motoda (Ed.), Active Mining, no. 79 in Frontiers in Artificial Intelligence and Applications, IOS Press, Amsterdam, 2002. [12] R. Miniati, F. Frosini, G. Cecconi, F. Dori, G. Gentili, Development of sustainable models for technology evaluation in hospital, Technol Health Care.
1037
ScienceDirect Procedia Computer Science 91 (2016) 1028 – 1037
Information Technology and Quantitative Management (ITQM 2016)
Data Mining oriented Software Quality Estimation Shusaku Tsumotoa,1 , Shoji Hiranoa a Department
of Medical Informatics, Faculty of Medicine, Shimane University, 89-1 Enya-cho, Izumo 693-8501 Japan
Abstract Clinical environment is very complex, and flexible and adaptive service improvement is crucial in maintaining quality of medical care. Thus, incremental update of software service in a hospital information system (HIS) and its evaluation is important. This paper introduces an active mining process for development of a an embedded software in which service logs stored in HIS are used to calculate the test statistics for evaluation on the effect of introduction of a new alarming service for clinical practice. are used to measure the differences The results show that proposed method is useful to evaluate the system performance in a real clinical environment. c 2016 2016Published The Authors. Published Elsevier B.V. © by Elsevier B.V. by This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Selection and/or peer-review under responsibility of ITQM2016. Peer-review under responsibility of the Organizing Committee of ITQM 2016
Keywords: software quality estimation process; active mining; hospital information system
1. Introduction Hospital information system (HIS)[1] has been introduced in a large-scale hospital and computerized clinical process. Stored data include all the results of clinical actions in a hospital, including laboratory examinations, physical examinations and clinical decisions. Since all the stored information are easily searched and visualized, the quality of hospital service depends on the quality of HIS. HIS improves the traceability of medical services: all the clinical actions are recorded with issued and executed time. Secondly, the overvall computerized service supports not only medical decisions and hospital management by using the stored data. However, these subjects have been rarely examined in a quantitative way. With respect to the first improvement, Hbner-Bloder and Ammenwerth[2] use performance indicators calculated from a questionnaire to the patients and doctors and the changes are detected before and after HIS has been introduced, although they can only evaluate overall performance, which cannot detect the details of improvements. Concerning the latter observation, Anema et al[3] use the clinical indictors for the quality of surgical operations. Although these studies give considerable insights, the analysis is not based on data stored in HIS. Since HIS keeps the records of clinical actions, evaluation can be made by using stored data. Although Tsumoto et al. [4, 5] applied to data mining methods for this purpose, the results are very naive. Another important point is that HIS is not complete. Since clinical environments are complex, all the requirements cannot be retrieved from the clinical scend, and developement of HIS should be incremental. However, the effect of improvements is not so easy to evaluate and subjective studies have been mainly used for evaluation. ∗ Corresponding author. Tel.: +81-853-20-2171 ; fax: +81-853-20-2170. E-mail address: [email protected].
1877-0509 © 2016 Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Organizing Committee of ITQM 2016 doi:10.1016/j.procs.2016.07.141
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
This paper proposes an active-mining framework for evaluation of improved software as follows. First, a problem pointed out by medical staff is selected and related data is extracted from HIS. Second, applying data mining methods, the characteristics of the problem are analyzed. Then, medical staff interprets the results and discuss the solutions. Proposed solutions are implemented on a hospital information system and the system keeps tracks of the usage of the improved program. Then, finally, the service is evaluated by the analysis of stored service logs. We focused on one practical problem for evaluation of the problem in our university hospital illustrated the effectiveness of this framework, based on active mining cycle[6]. 2. Motivation 2.1. Architecture of HIS HIS [1] follows service-oriented architecture and keeps records of all the clinical actions, including laboratory examinations, physical examinations, medical decision making, nursing actions, and so on. Incident or accident reports are not exception: they are also stored in HIS as clinical databases. The important characteristics of HIS is that the stored data includes spatial and temporal information: when are where orders are issued and executed. For example, Figure 1 shows the architecture of the HIS in Shimane University Hospital. HIS can be viewed Infection Control Support ing System
ECG Filing System Radiogram Reporting System
Electrophysiologocial Examination Supporting System
Respiratory Function Filing System
Interface to Filming Equipments
Ultrasonic Filing System
Reference Image/Interpretation PACS
Blood Collection Prepartion
Pathological Examinations Reporting System
Clinical Laboratory Examination Managment
Clinical Trial Management system
Data Mining System
Radiology Information System
Pharmaceutical Dept Managment
Bacterial Test
Rehabilitation
Radiology Department
Regional Heathcare Supporting System
Blood Transfusion Management Syssten
Narcotic Drug Management
Endoscopic Image Filing System
Operation Department
Blood Sample
Blood Transfusion Department
Laboratory Examination Department
Drug administration Guidance Endoscope Dept
Pathology Department
Rehabilitation Department
Pharmaceutical Department
Analysis for Managment
Medicines Stock Management System
Medical Informatics
Hospital Information System
Logistics Supporting System Incident Reporting System
Nutritional Management Department
㪤㪼㪻㫀㪺㪸㫃㩷㪠㫅㪽㫆㫉㫄㪸㫋㫀㪺㫊㩷 Division of Medical Informatics Order-Entry System Hospitalize/Release Food Service Reservation
Physiological Exam Blood Transfusion Radiology Labo Exam
Prescription Injection
Rehabilitation Medical Procedure
Pathology
DPC
Disease Type
Operation
Electronic Patient Records
Medical Professions Division
Food Service Supporting System
Nursing Support System Medical Service Payment System
Patient Records Authorization EPR viwer Patient Selection Drawing Memo
Patient Registration Card Bedside Entry Medical Document
Outpatient Clinic Management
Scanning System Discharge Summary
Datawarehouse for Medical Business
Fig. 1. Hospital Information System
as a large-scale network, through wich all the clinical information are exchanged as a cyberspace embedded in a hospital, keeping track of temporal information. By using service log, medical service can be visualized. Tsumoto et al. investigates the principal factor related with length of stay and medical payment by applying univariate and multivariate statistical analysis to data of Chiba Universiy Hospital [7, 8]. However, the most important part of stored informartion is “temporal”. Focusing on this aspects, Tsumoto et al. [9, 10] applied temporal data mining methods to capture the characteristics of the university hospital. 2.2. Order in HIS The basic information unit of HIS is an “order”, since HIS was firstly introduced as a order-entry system. For example, prescription is an order from a doctor to a pharmacist who will prepare for the pills for the therapy. Workflow of a prescription order will be shown as:
1029
1030
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
1. 2. 3. 4. 5. 6. 7.
Outpatient Clinic A prescription given from a doctor to a patient The patient bring it to medical payment department The patient bring it to pharmaceutical department Execution of order in pharmacist office Delivery of prescribed medication Payment
Figure 2 illustrates the information flow of prescription order, which can be implemented by UML. The second to fourth steps can be viewed as information propagation through the network. HIS keeps all the executed process of the prescription based on the information flow shown in Figure 2 as service logs.
Clinic
Pharmaceutical Department Prescription (Issued Order)
Pills Execution
Patient
Bill Order Medical Payment
Fig. 2. Workflow of Prescription Order
2.3. Patient Flow at Outpatient Clinic Figure 3 below shows a patient’s direction in a hospital for an outpatient clinic visit in Japan, which can be partially traced by the log of the hospital information system. First, a patient registers for their appointment at the reception desk. Then, the patient moves to a laboratory examination room and pre-ordered laboratory tests are performed. Next, the patient arrives at an examination room and waits for a physical examination by the doctor. During this process, the doctor will issue orders, such as prescriptions, further laboratory tests and other examinations, and reserve a time for the patient for his/her next visit. Finally, the patient will go back to the reception again to pay for services rendered, and the patient will be given a receipt and a paper for his/her future reservation. Figure 3 below illustrates patients movement in outpatient clinic. A patient first visits the reception desk, and then goes to laboratory examination room. Then, after laboratory examinations, he/she goes to the outpatient clinic and waits for calling from a doctor. When a doctor calls a patient, the doctor interviews the patient, starts physical examinations and evaluate all the examinations. Finally, the doctor issues prescription and reserve the next visit with reservation of laboratory examination.
Reception
Reception
Execution of Order (Laboratory Examination)
Issuance of Order Clinic (Physical Examination)
Label Printing Arrival at an examination room
(Prescription, Examinations, Reservation to Next Visit)
Start of Examination
Reception (Payment,Handout of Paper for Reservation)
End of Examination
Reserved Time
Fig. 3. Patient Flow in Outpatient Clinic
3. Method 3.1. Active Mining Process Motoda et al [11] proposed activing mining process in 2002, which is a comprehensive data mining and decision making process, as summarized in Figure 4. In the process, first we apply data mining methods to the
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
original data (active-user oriented mining). Then, the user evaluate the mining results, which may lead to a new discovery (active user reaction). Next, according to the users intepretation, new information gathering will be performed and the new stage for data collection (active information gathering) will be evoked. After enough data are stored, data mining process will be applied again as a active mining cycle.
3.2. Software Quality Estimation Process Based on the active mining process, we propose the fo f llowing software f development process as shown in Figure 5. First, data extracted from hospital information system is used to characterize a given problem where software development is important for a solution (information gathering). After temporal data mining methods are applied (data mining), the results are interpreted by medical staff and the solutions are discussed (active user reaction). Based on the discussions, new programs will be developed and service logs will be stored for its execution (active informat f ion gathering). Then, their performance f was evaluated using the service logs by using data mining methods (data mining). Data mining with Service Logs
Hypothesis Generation
Data Integration
Hypothesis V rification Ve
Software Implementation
Service Execution (Storage of New Logs)
Data in HIS Fig. 5. Software Quality Estimation Process
3.3. Experimental Setting and Quality t Evaluation In this paper, the data used for hypothesis generation is the chronological change of the number of clinical orders and waiting time. Analytical method of temporal data is based on multiscale matching method proposed in [9]. Concerning the meaning of chronological change of the number of orders and the analysis method, the reader could refer to [10].
1031
1032
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Experiments were conducted from fiscal year 2010 to 2012. In the ends of fiscal year 2010 and 2011, the new softwares were embedded into hospital information system and service logs are collected. Service logs which records the patient movement were extracted, and the time difference ff between visit to each division was calculated. Fiscal year 2010 gives the baseline, the first and second improvement was implemented in 2011 and 2012, repectively. Statistical analyses, including Kruskal-Wallis test were applied to statistics obtained from service logs, where R3-1-4 was used. 4. Experimental Results 4.1. First Cycle: Hypothesis Generation (Data Mining) Figure 6 shows a temporal chart of the number of orders on Tuesday in fiscal 2010. Vertical and horizonal axes show the number of orders and each timing slot, whose unit is one hour. The peak of each order are almost located between 9am to 5pm, corresponding to opening hours of outpatient clinic. The shape of the temporal curve characterize each clinical division, which can be captured by temporal data mining methods where temporal data mining methods, as discussed in [10]. For example, Figure 7 and 8 shows the temporal charts of hepatology
Fig. 6. Temporal Chart of Total Number of Orders (Fiscal Year 2010):Vertical axis and horizontal axis denote the service hours from 0 to 23 hours and the number of orders. Each point plots the number of orders during the time interval. For example, points on 9 of the vertical axis show the number of orders counted from 9:00 to 9:59.
Fig. 7. Temporal Chart of Number of Orders (Hepatology, Fiscal Year 2010)
and rheumatology in 2010. While the patterns of Figure 7 is very similar to Figure 6, Figure 8 is different ff from
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Prescription Examination Radiology Operation Transfusion Meal Pathology Injections Reservation Document Nursing Procedure Record Rehabilitation Admission
Fig. 8. Temporal Chart of Number of Orders (Rheumatology, Fiscal Year 2010)
those as follows. First, the peak of laboratory examination of rheumatology is in 9 to 10am, though those of others are located around 12pm. Secondly, records and reservation have the high numbers from 10am to 5pm, compared with other orders. Thus, the obtained results shows that a rhuematologist use a HIS interface in a different ff way from a hepatoligst whose workflow in the outpatient clinic reflects the ordinary pattern. This hypothesis will not only lead to the temporal change of the number orders, but also influence over service execution time for a rheumatologist. So, the next step is to measure the effect ff of the workflow change, which may be captured by temporal interval between these steps: the time difference ff between reception and execution of order, that between execution of order and its report, that between reporting and physical examination. Also, since each patient has a reserved timeslot, the temporal difference ff between reservation and actual examination is important, because this factor is related with the patients’ satisfaction. Table 1 shows the statistics of two divisions in 2010. The first column shows that ten minutes and 15 minutes are needed for laboratory examinations in the case of hepatology and rheumatology. It is notable that the medians are equal, which suggests that a significant number of patients of rheumatology waits longer than ones of hepatology. Table 1. Statistics of Waiting Time in 2010
Reception-Lab Lab-Lab Report LaboReport-Clinic Reservation-Clinic
Hepatology Average Median 10.48 4 25.22 20.0 96.83 77.0 48.45 31.0
Rhematology Average Median 14.83 4 19.0 17 95.58 77.0 51.82 48.0
4.2. First Cycle: Hypothesis Verification and Software Development The obtained results were shown to staff of laboratory and rheumatology, the following two points were found for Figure 8. First, during the peak of laboratory examinations, laboratory staff found missing orders and asked the doctors to reissue the orders. Then, secondly, although rheumatologists usually issued orders after their clinic has been finished, they sometimes forgot to book some orders because they left many orders not issued. Usually, the doctors should issue orders during physical examinations, but rheumatologists postpone booking due to their time constratins. The statistics shows they examined about 6 to 7 patients per hour, totally 30-35 patients on Tuesday. In the avarage, seven of them have missing orders in total. So the rate for forgetting orders can be estimated as 20-23%, This is the main reason why the patients wait for longer. Thus, the second discovery is the cause of the first discovery, which should be improved as a first step.
1033
1034
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Table 2. Statistics of Waiting Time of Hepatology from 2010 to 2013
(min)
Reception to Laboratory
Laborary Examination
2010 2011 2012 2013
10.17 ± 19.31 10.48 ± 20.00 8.94 ± 22.38 7.79 ± 20.29
23.95 ± 17.53 25.21 ± 17.25 27.23 ± 14.98 26.04 ± 13.60
Reporting to Inspection Reporting to Inspection 85.60 ± 66.19 96.83 ± 79.18 92.14 ± 73.68 95.57 ± 73.85
Reservation to Inspection 36.16 ± 58.00 48.45 ± 76.20 44.10 ± 67.37 43.94 ± 65.67
Figure 9 shows the workflows of outpatient clinic of hepatology and rheumatology after the discussion, whose manifestations are shown in Figure 7 and 8 , respectively. Thus, if the forget errors are improved, the temporal chart of laboratory examinations of rheumatology may be near to that of hepatology. Also, the changes can be measured as improvements of waiting time.
Hepatology Reception
Execution of Order (Laboratory Examination)
Clinic
Issuance of Order
Execution of Order
Clinic
Issuance of Order?
Rheumatology Reception
Issuance of Order Fig. 9. Workflow of Outpatient Clinic of Hepatology and Rheumatology Confirmed by Interview with Doctors
In fiscal year 2011, the alarm was implemented for the solution of the second problem: when a doctor wrote down the comment in the reservation confirmation, such as laboratory examinations needed, but left the orders not issued during the visit and planned to issue after all the clinic had been finished. However, the doctors forgot to issue them. Then, during the next visit the missing orders were found at laboratory examination room, and the laboratory staff would call the doctors and ask them to issue an order. To improve the situation, the first step was to introduce a alerming system for the discrepancy between reservation confirmation and issued order: if it detected before they closed a window for the patient, alerm will be evoked. 4.3. Second Cycle One year trial showed that waiting time has a small improvement. So, we interviewed the rheumatologist and prepared for the next solution: implementation of management of the orders when they finished their clinic and before they logged off in fiscal year 2012: When the doctor is going to log off, the management screen of orders were poped up to check all the forgot orders where the doctors issue orders from that screen. 4.4. Evaluation Results of First and Second Cycles 4.4.1. Change from 2010 to 2013 (Hepatology) Figure 10 shows the change of waiting times from 2010 to 2013 in Hepatology. cTable 3 shows the results of Kruskal-Wallis in terms of the time interval between reception and execution of laboratory examinations, which reflects the results obtained from Figure 10 and Table 2.
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037 Averaged Time needed between Laboratory Exam and Reporting
26 Time 24
8
25
9
Waiting Time
10
27
11
Averaged Waiting Time between Reception and Laboratory Exam
n=4246
n=4148
n=4212
n=4013
n=4246
n=4148
n=4212
n=4013
2010fy
2011fy
2012fy
2013fy
2010fy
2011fy
2012fy
2013fy
Fiscal Year
Averaged Time needed between Laboratory Reporting and Examination by the Doctors
Averaged Waiting Time between Reserved Time and Examination by the Doctors
Time 35
85
40
90
Time
45
95
50
Fiscal Year
n=4246
n=4148
n=4212
n=4013
n=4246
n=4148
n=4212
n=4013
2010fy
2011fy
2012fy
2013fy
2010fy
2011fy
2012fy
2013fy
Fiscal Year
Fiscal Year
Fig. 10. Change of Hepatology in 2010 to 2013. Vertical and horizontal axes denote the fiscal year and time needed for executing services.
Table 3. Kruskal-Wallis Test (Reception and Execution of Laboratory Examinations (p-value))
2010 2011 2012
2010 −
2011 0.900 −
2012 0.031 0.003 −
2013 < 0.001 < 0.001 0.052
4.4.2. Change from 2010 to 2013 (Rheumatology) Figure 11 plots the transition from 2010 to 2013 in Rhuematology. Since the values are monotonically improving, both of the interface implementation was effective. ff Table 5 shows the results of Kruskal-Wallis test in terms of to the time interval between reception and execution of laboratory examinations. However, the difference ff between 2010 and 2011 is not statistically significant. The differences ff also show that the second improvement is much better than the first one also for rheumatology (Table 4). However, the other waiting times increased, so total waiting time were not changed. Thus, although patient satisfaction may have been improved for laboratory examination, overall satisfaction may not. 5. Discussion Although quantative evaluation of HIS is important with respect to improvements of medical service quality, empirical evaluation has just started as shown in [12]. The difficulty is due to hardness of automated data acquisition for evaluation. Usually, questionnaire of end-users are used for estimation of the system, which is frequently used for software evaluation in other fields. However, it should be pointed software services are now embedded into large-scale software and that histories of execution are automatically accumulated, which can be viewed as “big data”. Thus, the way how to use the stored big data is indispensable for the improvements in embedded
1035
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Averaged Time needed between Laboratory Exam and Reporting
Time
20.5
13 10
19.5
11
20.0
12
Waiting Time
14
21.5
15
22.0
16
22.5
Averaged Waiting Time between Reception and Laboratory Exam
21.0
1036
n=2660
n=2955
n=3880
n=3680
2010fy
2011fy
2012fy
2013fy
n=2660
n=2955
n=3880
n=3680
2010fy
2011fy
2012fy
2013fy
Fiscal Year
Averaged Time needed between Laboratory Reporting and Examination by the Doctors
Averaged Waiting Time between Reserved Time and Examination by the Doctors
50
95
55
60
Time
105 100
Time
110
65
115
70
Fiscal Year
n=2660
n=2955
n=3880
n=3680
n=2660
n=2955
n=3880
n=3680
2010fy
2011fy
2012fy
2013fy
2010fy
2011fy
2012fy
2013fy
Fiscal Year
Fiscal Year
Fig. 11. Change of Rheumatology in 2010 to 2013. Vertical and horizontal axes denote the fiscal year and time needed for executing services.
Table 4. Statistics of Waiting Time of Rheumatology from 2010 to 2013
(min)
Reception to Laboratory
2010 2011 2012 2013
14.83 ± 32.50 13.64 ± 29.64 11.16 ± 30.52 10.85 ± 28.52
Laborary Examination to Reporting 19.98 ± 13.67 21.34 ± 13.33 22.05 ± 11.08 22.01 ± 11.12
Reporting to Inspection
Reservation to Inspection
95.58 ± 84.60 96.16 ± 81.6 115.71 ± 101.70 101.83 ± 85.07
51.82 ± 46.21 51.08 ± 47.88 69.05 ± 76.70 54.68 ± 47.93
services. But, it should be pointed out that such data storage and analyis are based on “retrospective” platform, whose evidence is weaker than experimental designed scheme, called “prospective analysis”. Thus, “big data” analytics should be curated by such a data collection form. Thus, active mining process may give a nice platform prospective analysis because it can consider the effects ff of intervention in a comprehensive way. The experiments and their results shown in this paper can be viewed as the first achievement of active-mining oriented software evaluation process. 6. Conclusion This paper proposes the following active-mining centered software development process. First, a problem pointed out by medical staff is selected and related data is extracted from HIS. Second, applying data mining methods, the characteristics of the problem are analyzed. Then, the results obtained are interpreted by medical staff and the solutions will be discussed. Proposed solutions are implemented on a hospital information system and the system keeps tracks of the usage of the improved program. Then, finally, the service is evaluated by the analysis of stored service logs.
Shusaku Tsumoto and Shoji Hirano / Procedia Computer Science 91 (2016) 1028 – 1037
Table 5. Kruskal-Wallis Test (Reception and Execution of Laboratory Examinations (p-value))
2010 2011 2012
2010 −
2011 0.452 −
2012 < 0.001 0.0042 −
2013 < 0.001 0.0011 0.964
The process was evaluated in Shimane Unversity Hospital. Two divisions, hepatology and rheumatology were selected for comparison. The obtained mining results gave a hypothesis that the workflow of rheumatology is different from that of hepatology, which reflected the ordinary workflow in the outpatient clinics of the university hospital and caused the problem where experts forgot to issue the orders. The first step for solution was to implement an alarming: if a doctor input a comment on laboratory examination in a reservation sheet but has not yet issued an order before closing the file, an alert will be evoked. However, after one year trial, statistics showed a small improvement in waiting time: after the discussion, in fiscal year 2012, another alarming screen was implemented where all the forgotten orders for patients who visited that day would be displayed, and the doctors could go back to issue orders. The workflow and waiting time were improved after the second installation. The process, which can be viewed as application of active mining process, will give a new framework for quantitative evaluation of software development in hospital information system. Acknowledgements This research is supported by Grant-in-Aid for Scientific Research (B) 15H2750 from Japan Society for the Promotion of Science(JSPS). References [1] E. Shortliffe, J. Cimino (Eds.), Biomedical Informatics: Computer Applications in Health Care and Biomedicine, 3rd Edition, Springer, 2006. [2] G. Hubner-Bloder, E. Ammenwerth, Key performance indicators to benchmark hospital information systems - a delphi study, Methods Inf Med (6) (2009) 508518. [3] H. Anema, J. Kievit, C. Fischer, E. Steyerberg, N. Klazinga, Inuences of hospital information systems, indicator data collection and computation on reported dutch hospital performance indicator scores, BMC Health Serv Res. (2013) 212. [4] S. Tsumoto, Knowledge discovery in clinical databases and evaluation of discovered knowledge in outpatient clinic, Information Sciences (124) (2000) 125–137. [5] S. Tsumoto, G5: Data mining in medicine, in: W. Kloesgen, J. Zytkow (Eds.), Handbook of Data Mining and Knowledge Discovery, Oxford University Press, Oxford, 2001, pp. 798–807. [6] S. Tsumoto, T. Yamaguchi, M. Numao, H. Motoda (Eds.), Active Mining, Second International Workshop, AM 2003, Maebashi, Japan, October 28, 2003, Revised Selected Papers, Vol. 3430 of Lecture Notes in Computer Science, Springer, 2005. [7] Y. Tsumoto, S. Tsumoto, Exploratory univariate analysis on the characterization of a university hospital: A preliminary step to datamining-based hospital management using an exploratory univariate analysis of a university hospital, The Review of Socionetwork Strategies 4 (2) (2010) 47–63. [8] Y. Tsumoto, S. Tsumoto, Correlation and regression analysis for characterization of university hospital (submitted), The Review of Socionetwork Strategies 5 (2) (2011) 43–55. [9] S. Tsumoto, S. Hirano, Detection of risk factors using trajectory mining, J. Intell. Inf. Syst. 36 (3) (2011) 403–425. doi:10.1007/s10844009-0114-7. [10] S. Tsumoto, H. Iwata, S. Hirano, Y. Tsumoto, Similarity-based behavior and process mining of medical practices, Future Generation Comp. Syst. 33 (2014) 21–31. doi:10.1016/j.future.2013.10.014. [11] H. Motoda (Ed.), Active Mining, no. 79 in Frontiers in Artificial Intelligence and Applications, IOS Press, Amsterdam, 2002. [12] R. Miniati, F. Frosini, G. Cecconi, F. Dori, G. Gentili, Development of sustainable models for technology evaluation in hospital, Technol Health Care.
1037