Predicting return visits to the emergency department for pediatric patients: Applying supervised learning techniques to the Taiwan National Health Insurance Research Database

Computer Methods and Programs in Biomedicine 144 (2017) 105–112

Contents lists available at ScienceDirect

Computer Methods and Programs in Biomedicine journal homepage: www.elsevier.com/locate/cmpb

Predicting return visits to the emergency department for pediatric patients: Applying supervised learning techniques to the Taiwan National Health Insurance Research Database Ya-Han Hu a, Chun-Tien Tai a,b, Solomon Chih-Cheng Chen c,d, Hai-Wei Lee a, Sheng-Feng Sung e,∗ a

Department of Information Management and Institute of Healthcare Information Management, National Chung Cheng University, Chiayi County, Taiwan Chiayi Chang Gung Memorial Hospital, Chiayi County, Taiwan c Heng Chun Christian Hospital, Pingtung County, Taiwan d Department of Pediatrics, School of Medicine, Taipei Medical University, Taipei, Taiwan e Department of Internal Medicine, Ditmanson Medical Foundation Chia-Yi Christian Hospital, 539 Zhongxiao Rd., Chiayi City, 60002 Taiwan b

a r t i c l e

i n f o

Article history: Received 31 August 2016 Revised 26 January 2017 Accepted 24 March 2017

Keywords: Data mining Emergency department NHIRD Pediatric emergency care Return visit

a b s t r a c t Background and objective: Return visits (RVs) to the emergency department (ED) consume medical resources and may represent a patient safety issue. The occurrence of unexpected RVs is considered a performance indicator for ED care quality. Because children are susceptible to medical errors and utilize considerable ED resources, knowing the factors that affect RVs in pediatric patients helps improve the quality of pediatric emergency care. Methods: We collected data on visits made by patients aged ≤18 years to EDs from the National Health Insurance Research Database. The outcome of interest was a RV within 3 days of the initial visit. Potential factors were categorized into demographics, medical history, features of ED visits, physician characteristics, hospital characteristics, and treatment-seeking behavior. A multivariate logistic regression was used to identify independent predictors of RVs. We compared the performance of various data mining techniques, including Naïve Bayes, classification and regression tree (CART), random forest, and logistic regression, in predicting RVs. Finally, we developed a decision tree to stratify the risk of RVs. Results: Of 125,940 visits, 6,282 (5.0%) were followed by a RV within 3 days. Predictors of RVs included younger age, higher acuity, intravenous fluid, more examination types, complete blood count, consultation, lower hospital level, hospitalization within one week before the initial visit, frequent ED visits in the past one year, and visits made in Spring or on Saturdays. Patients with allergic diseases and those underwent ultrasound examination were less likely to return. Decision tree models performed better in predicting RVs in terms of area under curve. The decision tree constructed using the CART technique showed that the number of ED visits in the past one year, diagnosis category, testing of complete blood count, and age were important discriminators of risk of RVs. Conclusions: We identified several factors which are associated with RVs to the ED in pediatric patients. The knowledge of these factors may help assess risk of RVs in the ED and guide physicians to reevaluate and provide interventions to children belonging to the high risk groups before ED discharge. © 2017 Elsevier B.V. All rights reserved.

1. Introduction The emergency department (ED) is a critical medical resource in delivering emergency services. Clinicians must make rapid decisions to ensure patients are treated promptly. Medical errors are thus inevitable because of the complex processes and busy work environments within EDs [1,2]. ∗

Corresponding author. E-mail address: [email protected] (S.-F. Sung).

http://dx.doi.org/10.1016/j.cmpb.2017.03.022 0169-2607/© 2017 Elsevier B.V. All rights reserved.

Patients sometimes make return visits (RVs) to EDs, suggesting potential medical problems are unresolved during initial ED visits. Patients making RVs may contribute to ED overcrowding, which in turn increases waiting times to be seen, the average length of stay, and healthcare costs [3,4]. Therefore, the rate of unexpected RVs to EDs is considered a performance indicator of ED care quality [5,6]. A RV to the ED often means a greater consumption of medical resources and denotes a risk to patient safety for those with a potential for disease progression [4]. Patients making RVs may have a higher risk of death or hospitalization shortly after the

106

Y.-H. Hu et al. / Computer Methods and Programs in Biomedicine 144 (2017) 105–112

second visit [7]. Among patients who seek emergency services, children warrant particular attention because they belong to the group of high ED utilization [3,8] and require special care. Children may be at an increased risk of medical errors because of their need for age-specific and weight-based dosing. For example, about 17% of children returning to EDs within 72 h were admitted [4]. To improve the quality of emergency care, knowing what affects RVs is required before quality improvement interventions can be implemented. Although prior studies have identified factors that may be associated with RVs in children [9–18], most of them are single-site studies conducted in Western countries and might suffer from generalization problems. Only two studies have analyzed population-based data from Canada and the US respectively [15,17]. We previously explored the factors related to RVs based on administrative data from 6 branches of an urban hospital in Taiwan [19]. However, the study results might be undermined by an outcome assessment bias because not all RVs were captured [19]. Therefore, the aim of this study was to determine which factors are associated with RVs in children and to construct models to predict RVs based on a representative sample from a nationwide population-based insurance claims database. With an enrollment rate of more than 99% of the population and the universal coverage for inpatient, ambulatory, and emergency care, virtually all ED visits are recorded. We evaluated the prediction performance of various data mining techniques and developed a decision tree for identifying high risk patients who might return after ED visits. 2. Materials and methods 2.1. Data source In 1995, the Taiwan government launched the National Health Insurance (NHI) program, which covers nearly all the population of 23 million people. Therefore, the datasets sampled from the National Health Insurance Research Database (NHIRD), which was derived from NHI claims data, are nationwide representative and can mitigate generalization problems commonly seen in single-site studies. The study data was obtained from the Longitudinal Health Insurance Database 2005, an NHIRD subset that includes all the original claims data from 1997 to 2011 of one million NHI enrollees randomly sampled in 2005. The study protocol was approved by the Ditmanson Medical Foundation Chia-Yi Christian Hospital Institutional Review Board. An informed consent to participate in this study was exempted because all data have been de-identified. 2.2. Study sample We identified 457,428 visits made by patients aged ≤18 years to EDs between 1998 and 2009. During the period, NHI mandated a four-level triage system in Taiwan EDs. We excluded 96,632 visits that had a preceding ED visit for any reason within the past 3 days, or were made by patients who were subsequently hospitalized within 1 day after ED visit or declared dead at the ED because RVs were not possible for them. Another 36,836 visits were eliminated because of missing values, data inconsistencies, or errors. From the remaining 323,960 visits, we randomly selected one visit for each patient to avoid analyzing duplicates of patient data. Finally, a total of 125,940 visits comprised the study sample (index visits). 2.3. Outcome measure The outcome of interest was a RV for any reason within 3 calendar days of the index visit. Because the NHIRD does not contain

information on the time of a day when patients visited EDs, we were unable to measure time in hours. Index visits were classified into visits with RV and those without RV. 2.4. Variables of interest Factors that might influence the risk of RVs were categorized into six dimensions: demographics, personal medical history, features of ED visits, physician characteristics, hospital characteristics, and treatment-seeking behavior. Demographics included age and sex. Personal medical history was extracted from all the claims data from year 1997 until before the index visit and included history of allergic diseases (allergic conjunctivitis, allergic rhinitis, asthma, and atopic dermatitis), premature birth, developmental disorders (cerebral palsy, autism, intellectual disability, and attention deficit hyperactivity disorder), catastrophic illness, and chronic illness (e.g., diabetes, hypertension, cardiovascular disease, and renal disease). Features of ED visits involved triage level (from level 1 to 4 with level 1 representing the highest acuity), primary discharge diagnosis based on the International Classification of Diseases, Ninth Revision, Clinical Modification diagnosis codes, administration of intravenous fluid, types of examinations performed (urinalysis, stool routine, complete blood count, blood biochemistry, ultrasound examination, and radiological examination), the number of examination types, and whether consultation was obtained. Physician characteristics included age, sex, year of service, and specialty (board-certified pediatrician, board-certified emergency medicine specialist, or others). Hospital characteristics included hospital level and hospital region. Hospital level referred to the accreditation level (medical center, regional hospital, or district hospital). Hospital region indicated the geographical region of Taiwan where the hospital was located. Treatment-seeking behavior included the following: being hospitalized within 1 week before the index visit, the frequency of ED visits within 1 year before the index visit, the season and date of the index visit. We classified seasons into spring (February–April), summer (May–July), fall (August–October), and winter (November–January). The date of the index visit was categorized into weekdays (Monday–Friday), Saturdays, and Sundays. Public holidays when physician offices are generally closed were grouped with Sundays. 2.5. Statistical analysis Continuous variables were summarized as mean (SD) and categorical variables as counts and percentages. Comparisons between visits with and without RV were performed using χ 2 test for categorical variables and t tests for continuous variables. A multivariate logistic regression (LR) was used to evaluate the independent associations between variables of interest and the outcome. Because children might be transferred to other hospitals and these transfers normally happened on the day of the initial visit, we did a sensitivity test by repeating the multivariate logistic regression after excluding patients who returned on the same day. Two-tailed P values <0.05 were considered statistically significant. Statistical analyses were performed using Stata 13.1 (StataCorp, College Station, Texas). Because of imbalanced class distribution in the original dataset, we used under-sampling to create balanced datasets [20]. We randomly removed the majority class examples to achieve a 1:1 ratio of visits with and without RVs and repeated it 30 times by varying the seed values to obtain 30 data subsets. Four data mining techniques (Supplementary Methods), including Naïve Bayes (NB), classification and regression tree (CART), random forest (RF), and LR, were employed to construct classification models for predicting RVs. Specifically, we used the NaiveBayes, simpleCart, RandomForest, and SimpleLogistic modules in

Y.-H. Hu et al. / Computer Methods and Programs in Biomedicine 144 (2017) 105–112

Weka 3.6.14 open-source data mining software (www.cs.waikato. ac.nz/ml/weka) with default parameters. A 10-fold cross validation was used to estimate the sensitivity, specificity, accuracy and area under curve (AUC) of the classification models. The average of the estimates from these data subsets were compared using paired ttest. Model performance was evaluated using AUC. The AUC measures how well a model discriminates between examples at different levels of the outcome. An AUC value of 0.7 is considered the threshold of acceptable discrimination [21]. To generate a tool to flag at risk patients, a decision tree was derived from the original dataset using the CART technique. To simplify the generated tree, the minimum number of examples in each node was set to 2500. After the tree was built, the RV rate was calculated for each node in the tree. 3. Result 3.1. Characteristics of study sample Of the 125,940 visits, 6282 were followed by a RV within 3 days, yielding a RV rate of 5.0%. Among the 6282 RVs, 1834 (29.2%) were followed by admission to hospital within one day. Table 1 lists characteristics of the study sample. Most of the index visits were triaged as level 3, which means less urgent. Diseases of the respiratory system ranked first among the diagnostic categories followed by injury and poising. Almost all of the characteristics were significantly different between groups with and without RV except for sex, history of premature birth, developmental disorders and chronic illness, and proportions of undergoing stool routine, ultrasound examination, and radiological examination. 3.2. Predictors of RVs Table 2 presents the odds ratios from the multivariate LR. RVs to EDs were less likely in older patients, those with allergic diseases, and those who underwent ultrasound examination whereas patients who were triaged to a higher level, received intravenous fluid, had more examination types, underwent complete blood count, received consultation, had been hospitalized within one week, or had frequent ED visits within one year, were prone to RVs. Compared to those with diseases of the respiratory system, patients diagnosed as having disorders of sense organs, skin and subcutaneous tissue, musculoskeletal system and connective tissue, injury and poisoning, or symptoms, signs and ill-defined conditions had less RVs. Patients who visited a regional or a district hospital returned more often than those who visited a medical center. Compared to those visiting hospitals in northern Taiwan, patients who visited hospitals in central Taiwan were more likely to return whereas those who visited hospitals in southern Taiwan were less likely to return. Visits in spring or on Saturdays have the highest chance of incurring RVs. In the sensitivity analysis, 2601 patients returning to the ED on the same day of the index visit were excluded. The RV rate was 3.0% (3681/123,339). Supplementary Table shows the odds ratios from the multivariate LR. Disorders of skin and subcutaneous tissue, symptoms, signs and ill-defined conditions, number of examination types, stool routine, complete blood count, ultrasound examination, consultation, and hospital level were no longer independent predictors of RVs. On the other hand, urinalysis became an independent predictor of RVs.

107

(P < 0.001), CART (P = 0.010), and LR (P < 0.001) models. The AUCs of the RF, CART, and LR models were all above 0.7. On the other hand, the CART model attained the highest average accuracy and average sensitivity (all P < 0.001 compared to NB, RF, and LR models). 3.4. Decision tree for ED use Fig. 1 illustrates the decision tree constructed using the CART technique. The best and second best predictors to discriminate visits with RV from those without were the number of ED visits within one year. Other important discriminators included diagnostic category, testing of complete blood count, and patient age. The risk of RVs in each node was calculated and the leaf nodes were divided into three risk groups: high (>10%), medium (5%–10%), and low risk (<5%) groups. Among the 125,940 visits, 10.5% were categorized as high risk, 41.0% as medium risk, and 48.6% as low risk. The tree shows that patients at the highest risk of RVs were those with more than two ED visits within one year and undergoing testing of complete blood count. The AUC of the decision tree model was 0.718. 4. Discussion 4.1. Principal findings By analyzing a nationwide representative sample, we found a RV rate of 5.0%, which is within the range (2.5%–6.5%) in previous studies [4,12,14,17,19]. We identified several factors that were associated with RVs within 3 days of initial visits. Most of them remained significant in the sensitivity analysis. Knowing these factors may help assess the risk of RVs in EDs. Besides, they could potentially be used for risk adjustment when comparing rates of RVs between hospitals. Moreover, we created an easily understandable decision tree (Fig. 1). Its rules can be implemented in a clinical decision support system, which could be used to stratify the risk of RVs upon deciding whether to discharge a child. If children are identified as high risk for RVs, clinicians may, on one hand, reevaluate their need for admission. On the other hand, care management interventions can be deployed to ensure patients have access to necessary care outside the emergency settings. Interventions to prevent RVs could include computer-generated instructions, EDmade appointments, post-discharge telephone coaching, case management, and home environment intervention [22]. 4.2. Factors related to patients The finding that younger patients had a higher risk of RVs are in line with numerous previous studies [9,12–14,19]. Younger children might lack communication ability and are thus poor reporters of their symptoms and disease severity. In addition, parents of younger children, particularly first-time parents, are less experienced in caring for a sick child and tend to return to the ED with any worsening of symptoms [12]. Younger children with asthma had a greater likelihood of returning after acute ED asthma care [23]. Contradictorily, we found a lower risk of ED RVs in patients with a history of allergic diseases including asthma. Although the exact nature of this finding is unclear, several strategies used in the ED to manage asthma in children, such as preprinted order sheets and access to pediatricians, have successfully reduced RVs [24].

3.3. Performance of data mining techniques 4.3. Factors related to ED visits Table 3 shows the average of sensitivity, specificity, accuracy, and AUC values from the 30 data subsets. Per the AUC values, the RF model achieved the highest discrimination compared to NB

The rate of RVs was increased among children with a higher acuity. It is not surprising because patients with a more urgent

108

Y.-H. Hu et al. / Computer Methods and Programs in Biomedicine 144 (2017) 105–112

Table 1 Characteristics of the study sample. Characteristic Demographics Age, mean (SD), year Male Personal medical history Allergic diseases Premature birth Developmental disorders Catastrophic illness Chronic illness Features of ED visits Triage level Level 1 Level 2 Level 3 Level 4 Diagnostic category Infectious and parasitic diseases Endocrine, nutritional and metabolic, immunity, blood and blood-forming organs Mental disorders Nervous system Sense organs Circulatory system Respiratory system Digestive system Genitourinary system Skin and subcutaneous tissue Musculoskeletal system and connective tissue Congenital anomalies Symptoms, signs and ill-defined conditions Injury and poisoning External causes of morbidity and mortality Factors influencing health status and contact with health services Miscellaneous Intravenous fluid Number of examination types, mean (SD) Urinalysis Stool routine Complete blood count Blood biochemistry Ultrasound examination Radiological examination Consultation Physician characteristics Age, mean (SD), year Male Year of service Emergency medicine specialist Pediatrician Hospital characteristics Hospital level Medical center Regional hospital District hospital Hospital region Northern Central Southern Eastern and islands Treatment-seeking behavior Hospitalized within 1 week before index visit Frequency of ED visits within 1 year before index visit Season Spring (February–April) Summer (May–July) Fall (August–October) Winter (November–January) Date Monday–Friday Saturday Sunday or holiday Data are numbers (percentage) unless specified otherwise. ED, emergency department; SD, standard deviation.

Total (n = 125,940)

With RV (n = 6282)

Without RV (n = 119,658)

P

8.4 (5.5) 69,158 (54.91)

7.1 (5.7) 3462 (55.11)

8.4 (5.5) 65,696 (54.90)

<0.001 0.748

46,031 (36.55) 81 (0.06) 2722 (2.16) 1615 (1.28) 66,286 (52.63)

2017 (32.11) 5 (0.08) 124 (1.97) 98 (1.56) 3304 (52.59)

44,014 (36.78) 76 (0.06) 2598 (2.17) 1517 (1.27) 62,982 (52.64)

<0.001 0.624 0.295 0.045 0.950

5062 (4.02) 40,451 (32.12) 78,879 (62.63) 1548 (1.23)

399 (6.35) 1988 (31.65) 3833 (61.02) 62 (0.99)

4663 (3.90) 38,463 (32.14) 75,046 (62.72) 1486 (1.24)

4328 (3.44) 413 (0.33)

266 (4.23) 33 (0.53)

4062 (3.39) 380 (0.32)

260 (0.21) 399 (0.32) 3879 (3.08) 236 (0.19) 38,853 (30.85) 16,436 (13.05) 1434 (1.14) 3687 (2.93) 676 (0.54) 130 (0.10) 22,002 (17.47) 32,611 (25.89) 282 (0.22) 143 (0.11) 171 (0.14) 15,628 (12.41) 0.64 (0.92) 7585 (6.02) 647 (0.51) 19,836 (15.75) 11,058 (8.78) 652 (0.52) 40,353 (32.04) 3560 (2.83)

15 (0.24) 38 (0.60) 113 (1.90) 13 (0.21) 2369 (37.71) 945 (15.04) 71 (1.13) 154 (2.45) 12 (0.19) 7 (0.11) 1221 (19.44) 995 (15.84) 10 (0.16) 10 (0.16) 10 (0.16) 1221 (19.44) 0.80 (1.07) 530 (8.44) 35 (0.56) 1530 (24.36) 889 (14.15) 31 (0.49) 1986 (31.61) 279 (4.44)

245 (0.20) 361 (0.30) 3766 (3.15) 223 (0.19) 36,484 (30.49) 15,491 (12.95) 1363 (1.14) 3533 (2.95) 664 (0.55) 123 (0.10) 20,781 (17.37) 31,616 (26.42) 272 (0.23) 133 (0.11) 161 (0.13) 14,407 (12.04) 0.63 (0.91) 7055 (5.90) 612 (0.51) 18,306 (15.30) 10,169 (8.50) 621 (0.52) 38,367 (32.06) 3281 (2.74)

<0.001 <0.001 <0.001 0.621 <0.001 <0.001 0.784 0.457 <0.001

37.6 (7.3) 107,932 (85.70) 4.8 (4.9) 40,320 (32.02) 40,528 (32.18)

37.0 (7.3) 5215 (83.01) 4.4 (4.8) 1743 (27.75) 2279 (36.28)

37.6 (7.3) 102,717 (85.84) 4.8 (4.9) 38,577 (32.24) 38,249 (31.97)

<0.001 <0.001 <0.001 <0.001 <0.001

42,493 (33.74) 72,589 (57.64) 10,858 (8.62)

2078 (33.08) 3705 (58.98) 499 (7.94)

40,415 (33.78) 68,884 (57.57) 10,359 (8.66)

61,259 (48.64) 25,132 (19.96) 35,587 (28.26) 3962 (3.15)

3023 (48.12) 1346 (21.43) 1684 (26.81) 229 (3.65)

58,236 (48.67) 23,786 (19.88) 33,903 (28.33) 3733 (3.12)

724 (0.57) 1.03 (1.25)

91 (1.45) 1.82 (2.11)

633 (0.53) 0.99 (1.18)

32,035 (25.44) 30,181 (23.96) 30,162 (23.95) 33,562 (26.65)

1711 (27.24) 1502 (23.91) 1504 (23.94) 1565 (24.91)

30,324 (25.34) 28,679 (23.97) 28,658 (23.95) 31,997 (26.74)

71,318 (56.63) 16,997 (13.5) 37,625 (29.88)

3513 (55.92) 952 (15.15) 1817 (28.92)

67,805 (56.67) 16,045 (13.41) 35,808 (29.93)

<0.001

<0.001

0.040

0.001

<0.001 <0.001 0.001

<0.001

Y.-H. Hu et al. / Computer Methods and Programs in Biomedicine 144 (2017) 105–112 Table 3 Performance of the classification models.

Table 2 Estimates of OR using multivariate logistic regression analysis. Characteristic Demographics Age, per year Male Personal medical history Allergic diseases Premature birth Developmental disorders Catastrophic illness Chronic illness Features of ED visits Triage level Level 1 Level 2 Level 3 Level 4 Diagnostic category Infectious and parasitic diseases Endocrine, nutritional and metabolic, immunity, blood and blood-forming organs Mental disorders Nervous system Sense organs Circulatory system Respiratory system Digestive system Genitourinary system Skin and subcutaneous tissue Musculoskeletal system and connective tissue Congenital anomalies Symptoms, signs and ill-defined conditions Injury and poisoning External causes of morbidity and mortality Factors influencing health status and contact with health services Miscellaneous Intravenous fluid Number of examination types Urinalysis Stool routine Complete blood count Blood biochemistry Ultrasound examination Radiological examination Consultation Physician characteristics Age, per year Male Year of service Emergency medicine specialist Pediatrician Hospital characteristics Hospital level Medical center Regional hospital District hospital Hospital region Northern Central Southern Eastern and islands Treatment-seeking behavior Hospitalized within 1 week before index visit Frequency of ED visits within 1 year before index visit Season Spring (February–April) Summer (May–July) Fall (August–October) Winter (November–January) Date Monday–Friday Saturday Sunday or holiday ED, emergency department; OR, odds ratio. a Omitted because of collinearity.

OR (95% CI)

P

0.983 (0.977–0.988) 1.017 (0.965–1.072)

<0.001 0.521

0.822 0.871 0.928 0.806 0.953

(0.774–0.874) (0.342–2.222) (0.765–1.126) (0.639–1.016) (0.90 0–1.0 09)

< 0.001 0.773 0.450 0.068 0.097

1.834 (1.384–2.430) 1.357 (1.041–1.769) 1.221 (0.939–1.587) Reference

<0.001 0.024 0.137

0.971 (0.849–1.111) 0.951 (0.649–1.393)

0.668 0.797

0.805 (0.465–1.392) 1.245 (0.873–1.776) 0.492 (0.404–0.598) 0.724 (0.402–1.304) Reference 0.984 (0.907–1.067) 0.778 (0.603–1.003) 0.824 (0.695–0.978) 0.338 (0.190–0.602)

0.437 0.226 <0.001 0.282 0.690 0.053 0.027 <0.001

0.807 (0.365–1.784) 0.914 (0.847–0.987) 0.593 (0.542–0.649) 0.697 (0.369–1.317) 1.250 (0.645–2.424)

0.596 0.022 <0.001 0.266 0.508

0.807 (0.419–1.555) 1.376 (1.246–1.521) 1.068 (1.005–1.135) 1.052 (0.930–1.190) 0.701 (0.492–0.998) 1.188 (1.047–1.348) 1.095 (0.967–1.239) 0.553 (0.377–0.809) Omitteda 2.192 (1.919–2.504)

0.522 <0.001 0.033 0.418 0.049 0.007 0.152 0.002

0.998 0.944 0.999 0.953 0.966

0.553 0.122 0.854 0.159 0.283

(0.992–1.004) (0.878–1.015) (0.990–1.009) (0.891–1.019) (0.907–1.029)

<0.001

Classifier

Sensitivity Specificity Accuracy AUC

NB

CART

RF

LR

0.455 0.739 0.597 0.644

0.800 0.552 0.676 0.721

0.737 0.599 0.668 0.723

0.594 0.707 0.651 0.718

AUC, area under curve; CART, classification and regression tree; LR, logistic regression; NB, Naïve Bayes; RF, random forest.

condition are expected to return more frequently than those with a non-urgent condition [12,15]. More than a third of the patients were diagnosed with respiratory diseases and the proportion was even higher in patients with RV. These results are in agreement with previous studies [9,14]. With respiratory diseases as the reference group, patients diagnosed with certain disorders were less prone to ED RVs, such as disorders of sense organs, musculoskeletal system and connective tissue. It is probably because ED physicians might suggest these patients to receive care from specialists in outpatient departments, rendering such patients less likely to return. As for patients with injury and positioning, we speculate that those with high severity were hospitalized and thus excluded from the present study. Patients with minor injuries were probably referred to specialists in outpatient departments. Patients with gastroenteritis treated with intravenous rehydration were more likely to return than those without [18]. Although the present study did not focus on specific diseases, a similar result was observed. Patients usually receive intravenous fluid because they are unable to eat or dehydrated, or because medications must be administered intravenously. The need for intravenous fluid indicates a higher disease severity. Therefore, home care for these patients after hospital discharge is generally difficult, making them prone to returning. Similarly, an increased number of examination types and undergoing various examinations or consultation probably indicated serious conditions or uncertainty about the diagnosis and were thus associated with RVs. Interestingly, undergoing ultrasound examination decreased the risk of RVs. While the use of ultrasound has become standard in adult EDs, it also has the potential to improve children’s care in the pediatric ED [25].

4.4. Factors of RVs related to physicians and hospitals

Reference 1.151 (1.084–1.222) 1.178 (1.060–1.309)

<0.001 0.002

Reference 1.074 (1.003–1.151) 0.894 (0.838–0.953) 1.074 (0.932–1.239)

0.042 0.001 0.325

1.629 (1.288–2.061)

<0.001

1.385 (1.363–1.407)

<0.001

1.127 (1.049–1.211) 1.065 (0.989–1.147) 1.084 (1.006–1.167) Reference

0.001 0.098 0.033

Reference 1.113 (1.032–1.200) 0.978 (0.921–1.038)

109

0.006 0.457

Previous studies have suggested that the risk of RVs might be reduced if medical service is provided by senior physicians because of a decreased misdiagnosis rate [26,27]. However, we did not find an independent association between the year of service of physicians and RVs. Patients who visited hospitals other than medical centers tended to return. Fewer ED beds, no geriatric unit, no social worker in the ED, and fewer available hospital beds were independently associated with a shorter time to first ED RV in the elderly [28]. Medical centers in Taiwan are typically equipped with the most advanced medical technology and are resource rich. Therefore, patients discharged from the ED of a lower-level hospital might be transferred to or visit a medical center on their own because of unresolved medical problems. The nullification of significance of hospital level in the sensitivity analysis might also hint that some children were transferred on the same day of the index visit. Whether hospital region influences the risk of RVs has been inconsistent in previous studies. A Canadian study showed that

110

Y.-H. Hu et al. / Computer Methods and Programs in Biomedicine 144 (2017) 105–112

Fig. 1. Decision tree of predictors for RVs. Diagnosis group A includes endocrine, nutritional and metabolic, immunity, blood and blood-forming organs, mental disorders, circulatory system, respiratory system, digestive system, genitourinary system, skin and subcutaneous tissue, congenital anomalies, symptoms, signs and ill-defined conditions, and factors influencing health status and contact with health services. Diagnosis group B includes infectious and parasitic diseases, nervous system, sense organs, musculoskeletal system and connective tissue, injury and poisoning, external causes of morbidity and mortality, and miscellaneous. Diagnosis group C includes infectious and parasitic diseases, endocrine, nutritional and metabolic, immunity, blood and blood-forming organs, mental disorders, nervous system, respiratory system, digestive system, congenital anomalies, symptoms, signs and ill-defined conditions, and factors influencing health status and contact with health services. Diagnosis group D includes sense organs, circulatory system, genitourinary system, skin and subcutaneous tissue, musculoskeletal system and connective tissue, injury and poisoning, external causes of morbidity and mortality, and miscellaneous. CBC, complete blood count; ED, emergency department; RV, return visit.

hospital region (urban versus rural) did not predict ED return for pediatric mental health care [15]. On the contrary, a higher risk of pediatric ED RVs was found in the western region of the United States [17]. In our study, RV rates were significantly different across hospital regions, whether in the primary or the sensitivity analysis.

Access to alternative medical services outside hospitals might be different in different regions of Taiwan. Although the exact causes were unknown, it might have implications for future quality improvement efforts.

Y.-H. Hu et al. / Computer Methods and Programs in Biomedicine 144 (2017) 105–112

4.5. Factors related to treatment-seeking behavior Weekend visits to the ED were found to predict RVs in the elderly patients [28], probably because of limited ED resources, such as staff level and available hospital beds, during weekends. On the contrary, weekend and holiday visits were associated with a lower rate of RVs in pediatric patients [19]. In our study, we divided weekends and holidays into Saturdays and Sundays/holidays for analysis. We found that patients with Saturday visits were most likely to return. The reason might be that outpatient departments are closed on the day following a Saturday but are usually open on the day following a Sunday or a holiday. Whether the rates of RVs differ among seasons has been inconsistent in previous studies [10,12,14,19], considering the varied distribution of diseases across seasons. Although visits made in spring were associated with the highest chance of RVs in our study, the underlying mechanism awaits further investigation. Patients discharged from the hospital within 1 week before the index visit tended to return. This finding is in line with previous studies [17,29], indicating these patients had not yet completely recovered, either physically or psychologically, by the time they were discharged. The rate of RVs was significantly higher in patients with a higher frequency of ED visits in the past year. A similar result was found in adults [30]. As seen in adult counterparts, the main reason for using the ED by pediatric frequent ED users included lack of availability of their primary care physician and having chronic conditions [31]. 4.6. Choice of data mining techniques The RF and CART models significantly outperformed the LR model in terms of AUC and accuracy. It is possibly because we only included main effects in the LR model; large number of variables made it impractical to test for all possible interactions. Although LR models enable testing of statistical interactions among independent variables, it can be difficult to interpret the results, especially when more than two variables are assessed at a time. On the contrary, decision tree models, including RF and CART models, naturally incorporate interaction effects and are particularly well suited for investigating multilevel interactions [32,33]. Compared with the other classification models, the CART model is more suitable for stratifying risk groups because its decision rules are presented in a tree structure and are easy to use, either in printed form (Fig. 1) or embedded in a clinical decision support system. Although the multivariate LR analysis helps identify independent risk factors of RVs, it is too complex to use at the bedside even though a nomogram can be developed to graphically represent the prediction model. On the other hand, black-box approaches such as RF technique are not easy to understand by clinicians, not to mention the possibility that spurious correlations might go unnoticed because of a lack of insight in the model [34]. 4.7. Implications for clinical practice As several of the identified risk factors of RVs are related to higher disease severity or unresolved medical problems (for example, higher acuity, undergoing many types of examinations or consultation, hospitalized within one week before the index visit), and a substantial number of RVs (29.2%) were followed by admission compared with a 4.5% admission rate in Taiwan pediatric EDs [35], many RVs were indeed warranted. Clinicians should take a second thought before discharging a child with such characteristics from the ED. On the other hand, unavailability of primary care physicians might contribute to increased RVs in children visiting on Saturdays or having frequent ED visits. Interventions, such as

111

partnerships with primary care, telephone coaching, and thorough discharge instructions, could be considered. 4.8. Limitations The present study has several limitations. First, certain information was not available for analysis. For example, we could not obtain the exact time of ED entry or discharge, therefore, we had to investigate RVs within 3 calendar days instead of the more commonly used 72-h time frame. In addition, the length of stay in the ED was unavailable. Second, the actual reasons for returning were undetermined. We could not examine whether RVs were scheduled or unscheduled. Third, some selection bias was inevitable: newborns not yet insured were not included; children might be hospitalized via outpatient departments after the index visit. Fourth, certain important variables might be missing. We failed to include an exhaustive list of factors that might alter the risk of RVs. Finally, we did not explore the relationships between the diagnosis during the RV and the diagnosis and medical treatments during the index visit. For example, patients may return for a problem unrelated to the condition during the index visit. 5. Conclusion We identified several factors that are associated with RVs to the ED in children. Since this study is based on a national dataset, we believe the results could be representative and generalizable. The decision tree is easy to apply to assist clinicians in identifying patients with high risk for RVs, at whom reevaluation and interventions should be targeted before discharge. Some of the influential factors identified could be potentially used for risk-adjust RV rates for hospital comparison and may warrant in-depth evaluation in future studies. Acknowledgments This study is based in part on data from the National Health Insurance Research Database provided by the Bureau of National Health Insurance, Department of Health and managed by National Health Research Institutes. The interpretation and conclusions contained herein do not represent those of Bureau of National Health Insurance, Department of Health or National Health Research Institutes. This research was supported in part by the Ministry of Science and Technology (grant number MOST 104-2410-H-194-070MY3 and MOST 105-2314-B-705-001). We would like to thank Ms. Li-Ying Sung for English language editing. Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.cmpb.2017.03.022. References [1] D.T. Risser, M.M. Rice, M.L. Salisbury, R. Simon, G.D. Jay, S.D. Berns, The potential for improved teamwork to reduce medical errors in the emergency department. The MedTeams Research Consortium, Ann. Emerg. Med. 34 (1999) 373–383. [2] J. Fordyce, F.S.J. Blank, P. Pekow, H.A. Smithline, G. Ritter, S. Gehlbach, et al., Errors in a busy emergency department, Ann. Emerg. Med. 42 (2003) 324–333. [3] M.A. Hostetler, S. Mace, K. Brown, J. Finkler, D. Hernandez, S.E. Krug, et al., Emergency department overcrowding and children, Pediatr. Emerg. Care 23 (2007) 507–515. [4] R.D. Goldman, A. Kapoor, S. Mehta, Children admitted to the hospital after returning to the emergency department within 72 hours, Pediatr. Emerg. Care 27 (2011) 808–811.

112

Y.-H. Hu et al. / Computer Methods and Programs in Biomedicine 144 (2017) 105–112

[5] G.R. Hung, D. Chalut, A consensus-established set of important indicators of pediatric emergency department performance, Pediatr. Emerg. Care 24 (2008) 9–15. [6] C.R. Trivedy, M.W. Cooke, Unscheduled return visits (URV) in adults to the emergency department (ED): a rapid evidence assessment policy review, Emerg. Med. J. 32 (2015) 324–329. [7] G. Sauvin, Y. Freund, K. Saïdi, B. Riou, P. Hausfater, Unscheduled return visits to the emergency department: consequences for triage, Acad. Emerg. Med. 20 (2013) 33–39. [8] X. Luo, G. Liu, K. Frush, L.A. Hey, Children’s health insurance status and emergency department utilization in the United States, Pediatrics 112 (2003) 314–319. [9] D.R. Zimmerman, K.A. McCarten-Gibbs, D.H. DeNoble, C. Borger, J. Fleming, M. Hsieh, et al., Repeat pediatric visits to a general emergency department, Ann. Emerg. Med. 28 (1996) 467–473. [10] E.A. Alessandrini, J.M. Lavelle, S.M. Grenfell, C.R. Jacobstein, K.N. Shaw, Return visits to a pediatric emergency department, Pediatr. Emerg. Care 20 (2004) 166–171. [11] C.R. Jacobstein, E.A. Alessandrini, J.M. Lavelle, K.N. Shaw, Unscheduled revisits to a pediatric emergency department: risk factors for children with fever or infection-related complaints, Pediatr. Emerg. Care 21 (2005) 816–821. [12] R.D. Goldman, M. Ong, A. Macpherson, Unscheduled return visits to the pediatric emergency department-one-year experience, Pediatr. Emerg. Care 22 (2006) 545–549. [13] K. LeDuc, H. Rosebrook, M. Rannie, D. Gao, Pediatric emergency department recidivism: demographic characteristics and diagnostic predictors, J. Emerg. Nurs. 32 (2006) 131–138. [14] S. Costabel, E. Piccotti, M. Sartini, M. Magnani, P. Di Pietro, Return visits to the paediatric emergency department: first analysis in Italy, J. Prev. Med. Hyg. 49 (2008) 142–147. [15] A.S. Newton, S. Ali, D.W. Johnson, C. Haines, R.J. Rosychuk, R.A. Keaschuk, et al., Who comes back? Characteristics and predictors of return to emergency department services for pediatric mental health care, Acad. Emerg. Med. 17 (2010) 177–186. [16] A. Klein-Kremer, R.D. Goldman, Return visits to the emergency department among febrile children 3 to 36 months of age, Pediatr. Emerg. Care 27 (2011) 1126–1129. [17] C.S. Cho, D.J. Shapiro, M.D. Cabana, J.H. Maselli, A.L. Hersh, A national depiction of children with return visits to the emergency department within 72 hours, 20 01-20 07, Pediatr. Emerg. Care 28 (2012) 606–610. [18] S.B. Freedman, J.D. Thull-Freedman, M. Rumantir, E.G. Atenafu, D. Stephens, Emergency department revisits in children with gastroenteritis, J. Pediatr. Gastroenterol. Nutr. 57 (2013) 612–618. [19] S.-F. Sung, K.E. Liu, S.C.-C. Chen, C.-L. Lo, K.-C. Lin, Y.-H. Hu, Predicting factors and risk stratification for return visits to the emergency department within 72 hours in pediatric patients, Pediatr. Emerg. Care 31 (2015) 819–824.

[20] R. Longadge, S. Dongre, Class imbalance problem in data mining: Review, Int. J. Comput. Sci. Network 2 (2013). [21] M.P. LaValley, Logistic regression, Circulation 117 (2008) 2395–2399. [22] Q.K. Tran, J.D. Bayram, R.T. Boonyasai, M.A. Case, C. Connor, D. Doggett, et al., Pediatric emergency department return: a literature review of risk factors and interventions, Pediatr. Emerg. Care 32 (2016) 570–577. [23] C.M. Walsh-Kelly, K.J. Kelly, A.L. Drendel, L. Grabowski, E.M. Kuhn, Emergency department revisits for pediatric acute asthma exacerbations: association of factors identified in an emergency department asthma tracking system, Pediatr. Emerg. Care 24 (2008) 505–510. [24] A. Guttmann, B. Zagorski, P.C. Austin, M. Schull, A. Razzaq, T. To, et al., Effectiveness of emergency department asthma management strategies on return visits in children: a population-based study, Pediatrics 120 (2007) e1402–e1410. [25] A. Abo, K. Kelley, N. Kuppermann, S. Cusick, Ultrasound equipment for the pediatric emergency department: a review, Pediatr. Emerg. Care 27 (2011) 220–226. [26] C.P. Ng, C.H. Chung, An analysis of unscheduled return visits to the accident and emergency department of a general public hospital, Hong Kong J. Emerg. Med. 10 (2003) 153–161. [27] S.-C. Hung, G. Chew, C.-T. Kong, C.-T. Hsiao, S.-J. Liaw, Unplanned emergency department revisits within 72 hours, J. Emerg. Med. Taiwan 6 (2004) 230–238. [28] J. McCusker, R. Ionescu-Ittu, A. Ciampi, A. Vadeboncoeur, D. Roberge, D. Larouche, et al., Hospital characteristics and emergency department care of older patients are associated with return visits, Acad. Emerg. Med. 14 (2007) 426–433. [29] C.W. Burt, L.F. McCaig, A.E. Simon, Emergency department visits by persons recently discharged from U.S. hospitals, Natl. Health Stat. Report. No. 6, Hyattsville, MD, National Center for Health Statistics (2008). [30] G. Di Giuseppe, R. Abbate, L. Albano, P. Marinelli, I.F. Angelillo, Collaborative Research Group, Characteristics of patients returning to emergency departments in Naples, Italy, BMC Health Serv. Res. 8 (2008) 97. [31] E. LaCalle, E. Rabin, Frequent users of emergency departments: the myths, the data, and the policy implications, Ann. Emerg. Med. 56 (2010) 42–48. [32] S.C. Lemon, J. Roy, M.A. Clark, P.D. Friedmann, W. Rakowski, Classification and regression tree analysis in public health: methodological review and comparison with logistic regression, Ann. Behav. Med. 26 (2003) 172–181. [33] S. Poole, S. Grannis, N.H. Shah, Predicting Emergency Department Visits, AMIA Jt. Summits Transl. Sci. Proc. 2016 (2016) 438–445. [34] A. Feelders, H. Daniels, M. Holsheimer, Methodological and practical aspects of data mining, Inf. Manage. 37 (20 0 0) 271–281. [35] M.-J. Jeng, Y.-S. Lee, P.-C. Tsao, C.-F. Yang, Y.-C. Luo, W.-J. Soong, A 10-year population-based nationwide descriptive analysis of pediatric emergency care, BMC Pediatr. 14 (2014) 100.