- Email: [email protected]
Relationship between number of health problems addressed during a primary care patient visit and clinician workload
Applied Ergonomics 84 (2020) 103035
Contents lists available at ScienceDirect
Applied Ergonomics journal homepage: http://www.elsevier.com/locate/apergo
Relationship between number of health problems addressed during a primary care patient visit and clinician workload Jonathan L. Temte a, *, John W. Beasley a, c, Richard J. Holden b, Ben-Tzion Karsh c, 1, Beth Potter a, Paul Smith a, Peggy O’Halloran d a
University of Wisconsin School of Medicine and Public Health, Department of Family Medicine and Community Health, 1100 Delaplaine Court, Madison, WI, 53715, USA Indiana University School of Medicine, Department of Medicine 545 Barnhill Dr., Emerson Hall 305, Indianapolis, IN, 46202, USA c University of Wisconsin, Department of Industrial and Systems Engineering, 1415 Engineering Drive, Madison, WI, 53706, USA d Eau Claire City-County Health Department, 720 2nd Ave, Eau Claire, WI, 54703, USA b
A R T I C L E I N F O
A B S T R A C T
Introduction: Primary care is complex due to multiple health problems being addressed in each patient visit. Little is known about the effect of the number of problems per encounter (NPPE) on the resulting clinician workload (CWL), as measured using the National Aeronautics and Space Administration Task Load Index (NASA-TLX). Methods: We evaluated the relationship between NPPE and CWL across 608 adult patient visits, conducted by 31 clinicians, using hierarchical linear regression. Clinicians were interviewed about outlier visits to identify reasons for higher or lower than expected CWL. Results: Mean NPPE was 3.30 � 2.0 (sd) and CWL was 47.6 � 18.4 from a maximum of 100. Mental demand, time demand and effort accounted for 71.5% of CWL. After adjustment for confounders, each additional problem increased CWL by 3.9 points (P < 0.001). Patient, problem, environmental and patient-physician relationship factors were qualitatively identified from interviews as moderators of this effect. Conclusion: CWL is positively related to NPPE. Several modifiable factors may enhance or mitigate this effect. Our findings have implications for using a Human Factors (HF) approach to managing CWL.
1. Introduction The provision of primary medical care is a highly complex process (Wetterneck et al., 2011; Beasley et al., 2011; Holman et al., 2016) for which measurements of complexity have been proposed (Katerndahl et al., 2010). As compared to sub-specialty care, in which a single health problem within a single organ system is typically addressed, primary care spans the spectrum from disease prevention to end-stage disease and palliative care, from acute care to chronic disease management, and across all organ systems, age ranges and genders (Donaldson et al., 1996). Consequently, multiple and often unrelated and competing problems are bundled together within the context of a single visit (Beasley et al., 2004; Stange et al., 1998; Flocke et al., 2001; Coco,
2009). Across 34 h of direct patient care (White, 2012), the average family physician provides 84 outpatient visits each week (American Academy of Family Physicians, 2018), with each visit involving, on average, the evaluation and management of three or more problems (Beasley et al., 2004), thus resulting in more than 256 distinct problems per week. A problem is defined as a symptom, disease, abnormality (e.g. x-ray), psychological or social issue about which the clinician gathers data and makes a decision. Problems can include an acute health-related issue (e. g., sore throat), a chronic disease management issue (e.g., control of hypertension), a wellness issue (e.g., arranging a colonoscopy), or a psychosocial issue (e.g., concern about a spouse’s alcohol consumption), among others. This estimate does not take into account the additional
Abbreviations: CWL, clinician workload; EHR, electronic health record; NASA-TLX, National Aeronautics and Space Administration Task Load Index; NPPE, number of problems per encounter; WREN, Wisconsin Research and Education Network. * Corresponding author. Family Medicine and Community Health University of Wisconsin School of Medicine and Public Health, 1100 Delaplaine Court, Madison, WI, 53715, USA. E-mail addresses: [email protected] (J.L. Temte), [email protected] (B.-T. Karsh). 1 Deceased. https://doi.org/10.1016/j.apergo.2019.103035 Received 14 December 2018; Received in revised form 9 July 2019; Accepted 13 December 2019 0003-6870/© 2019 Elsevier Ltd. All rights reserved.
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
2018) practices that were selected to form a diverse and representative set of primary care sites, which included two urban and two rural set tings. After obtaining in-person informed consent, we trained clinicians to record data on routine adult patient visits over a four-week study period. Each clinician was instructed to provide data for 20 patient visits in order to achieve the initial target sample size of 600 clinical en counters. Data were collected between August 2007 and March 2008; whereas electronic health records (EHR) had been implemented at two sites, order entry using EHR—a time consuming task for the clin ician—had not yet commenced at this time. Clinic days were divided into early, mid, and late morning sessions and early, mid, and late afternoon sessions. On any given day, the cli nician’s medical assistant selected a patient from a randomly assigned time period. This quasi-randomization process was used to provide a representative patient population. No incentives were used for recruit ment. The only stipulations for recruitment were that the patient be an adult (age 18 or greater), not be a prisoner or institutionalized, and be free of significant impairment in decision-making capacity. No other inclusion/exclusion criteria were used so as to mirror the types of pa tients typically seen by primary care clinicians. The study protocol was approved by the University of Wisconsin Health Sciences Institutional Review Board. Following patient recruitment and obtaining informed consent by a trained clinical staff member, a medical assistant or nurse collected basic demographic data. At the time of rooming and before any contact with the study clinicians, patients identified the number of problems that they wanted to have addressed during the forthcoming encounter. Following the visit, the patient recorded the extent to which the clinician addressed their concerns using a 7-point Likert scale. The clinician did not have access to this information at any time. At the conclusion of the patient visit, the clinician completed three instruments to characterize the clinical encounter. The number of problems per encounter (NPPE) was recorded by the clinician who identified both the number and types of health related problems addressed during the patient visit using a form modified from the study of Beasley et al. (2004) A copy of the corresponding clinical encounter note was de-identified, labeled with the clinician’s ID and the encounter number, and attached with the study forms. This was available for reference if the clinician recorded problem log required clarification. CWL for the encounter was measured immediately following the visit using the NASA-TLX (Hart et al., 1988). The NASA-TLX is a weighted combination of six dimensions of work (mental demand, physical de mand, time demand, performance, effort and frustration) that reliably
burden imposed by inpatient care, home and nursing home visits, and management of problems by telephone, electronic communication and through intermediary clinical staff (Baron, 2010; Arndt et al., 2017). There has been comment on the impossible task of fully adhering to chronic disease guidelines within this severely limited timeframe (Østbye et al., 2005). Moreover, there has been recognition that critical decisions about patient care require high mental acuity (Helmreich, 2000; Young et al., 2008). Little attention, however, has been paid to the additive effect of managing multiple problems on clinician workload (CWL). Given the relationship between CWL and the likelihood of errors (Vidulich et al., 2012), a positive linear association between the number of problems and CWL would imply each problem a clinician evaluates and manages during a patient visit might increase the potential for medical error. We hypothesized that a relationship exists linking the number of problems managed during a single clinical visit and the resultant CWL for that visit. Furthermore, we hypothesized, but did not test in this study, that CWL is related to visit outcomes such as quality of care, medical error, and clinician burnout. The overall conceptual framework is illustrated in Fig. 1 and draws from prior models and empirical studies of CWL in healthcare (Holden et al., 2010, 2011; Carayon and Gurses, 2005). In this study, we used a validated instrument to measure the CWL of a diverse set of primary care clinicians within four practices affiliated with an established practice-based research network, the Wisconsin Research and Education Network (WREN) (Temte and Johnson, 2004; Wisconsin Research and Education Network, 2018). We examined whether one component of complexity—an increase in the number of problems a clinician addresses during a single patient visit—was associated with an increase in CWL as measured by the National Aeronautics and Space Administration-Task Load Index (NASA-TLX) (Hart et al., 1988; Byrne, 2011). We also assessed the relationship between CWL and other clinic, clinician, and patient visit characteristics. Finally, we employed a qualitative analysis of outlying observations to collect additional infor mation about the relationship between the number of problems addressed during a patient visit and resulting CWL. 2. Methods 2.1. Data We recruited thirty-one primary care clinicians from four WREN (Temte and Johnson, 2004; Wisconsin Research and Education Network,
Fig. 1. Simplified model of clinician workload in primary care, which may serve as a conceptual basis for future evaluations. This study tested components within the grey box. 2
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
measures workload (Hart et al., 1988; Byrne, 2011; Rubio et al., 2004). Moreover, this is a validated and commonly used instrument in human factors engineering (Karsh et al., 2006; Carayon, 2006). Although the NASA-TLX has had some use in health care settings, including emer gency medicine (France et al., 2005), telemedicine (Boultinghouse et al., 2007) and anesthesia (Byrne, 2011; Leedal and Smith, 2005; Weinger et al., 2004; Gaba and Lee, 1990), it had not been previously utilized in primary care settings. Finally, using three separate 7-point Likert scales, clinicians completed a form that assessed the extent to which they felt they addressed patient concerns and the estimated likelihood of possible omission and/or error (data for visit completeness, omissions, and errors were not used in this current assessment). In pilot tests, the total time required for all clinician instrument completion was less than 2 min. We purposely collected data for no more than one encounter per half day so as not to contribute additionally to clinician workload.
3. Results 3.1. Clinic and clinician characteristics Thirty-one clinicians from four WREN-affiliated primary care clinics participated in this study. Table 1 shows the characteristics of the participating clinics and clinicians, and describes the study patient en counters. Three of the clinics were located in Wisconsin; the fourth was located in Northeast Iowa on the Wisconsin border. Two of the sites were urban (17 clinicians) and two were rural (14 clinicians). Both urban practices were actively utilizing electronic medical record systems. Physicians, physician assistants, and nurse practitioners participated in the study, representing family medicine and internal medicine practices. Most participants were engaged in full-time practice and the clinicians tended to be experienced. On average, 15.5 years had elapsed since completion of training (median ¼ 14 years; range: 1–32 years) and the clinicians had been practicing at their current clinical sites for an average of 12.9 years (median ¼ 13.5 years; range: 1–28 years).
2.2. Analysis
3.2. Patient encounters
We initially examined associations between CWL and NPPE and other potential predictors of CWL using Pearson correlation coefficients and plotted the bivariate relationship between NPPE and CWL. We then used a hierarchical linear regression model (SAS 9.1) to assess the relationship between NPPE and CWL, accounting for the correlation of patients within clinicians at each of the four participating clinics. We accounted for the correlation of patients within clinician, and clinician within clinic by including clinicians within clinics as a random effect in the model. As is commonly observed, a single random effect in the final models accounted for the intra-group correlation. Finally, we included clinic, clinician, and patient characteristics in the model to identify other predictors of CWL and adjust the NPPE-CWL association. We excluded some potential confounders (practice: rural vs. urban; type and EHR use; patient: gender, day and time of appointment; clinician: gender, years out of training) from the multivariate analysis due to issues of collin earity or a lack of a significant correlation with CWL. We used Bayesian information criterion (BIC) to allow for model selection among a set of models, wherein the lowest BIC was preferred. BIC approaches over fitting of a model by introducing a penalty term for the number of pa rameters in the model.
We collected data from 608 patient encounters conducted by 31 study clinicians (mean ¼ 19.6 encounters per clinician; median ¼ 20) from August 2007 through March 2008 (Table 1). Complete data were available for 598 (98.4%) encounters. Study visits occurred more commonly on Tuesdays than expected and less commonly on Fridays than expected (X2 ¼ 10.3; df ¼ 4; P ¼ 0.03). Early afternoon visits were under-represented and mid-afternoon visits were over-represented in the final sample (X2 ¼ 18.0; df ¼ 5; P ¼ 0.003). Overall, however, there was good representation of encounters—from 8 to 29 visits—in each of 30 potential time slots. Table 1 Practice, clinician, and patient encounter characteristics. Practice Characteristics (n ¼ 4) Setting
Rural Urban Practice Type Family Medicine Internal and Family Medicine EHR in Use Yes No Clinician Characteristics (n ¼ 31) Gender Male Female Degree MD/DO PA/NP Schedule Part time Full time Years since completing training Years at practice site Patient Encounter Characteristics (n ¼ 608) Number of Problems 1–12 Patient Age 18–90 years
2.3. Qualitative assessment To assess important qualitative characteristics, we conducted brief clinician interviews on a subset of visits (n ¼ 34; 5.6% of all study visits) that had unexpectedly high or low CWLs relative to the number of problems. Eighteen of these events were selected based on an absolute standardized residual of 2.0 or greater from the clinician-specific simple linear regression model relating CWL to NPPE. Sixteen additional included events did not meet the above statistical definition of an outlier but were notably displaced from the regression line in graphical plots of CWL vs. NPPE. For these 34 total encounters, the de-identified clinical note was faxed to the clinician for review, and a brief interview was conducted in person or by phone. Interviews were conducted only if the visit was clearly remembered by the clinician (all visits were remem bered). A list of open-ended questions was presented to the clinician pertaining to anything that stood out about the visit, factors that may have contributed to a greater or lesser CWL, the relationship between the clinician and the patient, and contributing factors of clinic setting or schedule. The interviewers kept detailed notes regarding the salient points of the interview. The notes were typed and provided to three clinically active family physician reviewers (JB, BP, JT) for assessment and identification of themes associated with high and low CWL. The final list of factors was constructed following joint review of each case and development of consensus.
Patient Gender Continuity Patient Status Day of Appointment
Time of Appointment Nature of Presenting Problem
3
Male Female Yes No Monday Tuesday Wednesday Thursday Friday Morning Afternoon Acute Problem Chronic Problem Well Care Other
2 2 3 1 2 2 19 (61.3%) 12 (38.7%) 24 (77.4%) 7 (22.6%) 5 (16.1%) 26 (83.9%) 15.5 � 8.2 (sd) years 12.9 � 8.2 (sd) years 3.3 � 2.0 (sd) 54.6 � 17.5 (sd) years 221 (36.5%) 384 (63.5%) 450 (84.0%) 86 (16.0%) 123 (21.1%) 136 (23.3%) 114 (19.6%) 121 (20.8%) 89 (15.3%) 275 (52.7%) 247 (47.3%) 255 (43.4%) 265 (45.1%) 58 (9.9%) 10 (1.6%)
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
0.217; p ¼ 0.240; Fig. 4).
At the start of the encounter, patients reported an average of 2.32 � 1.42 (sd) problems they wished to have addressed. After the visit, they overwhelmingly reported that all their problems were addressed during the visit (91% responded at the highest level on the 7-point Likert scale). Clinicians reported evaluating and managing from 1 to 12 problems during each encounter, with an average NPPE of 3.30 � 2.0 (sd) and a median of 3 problems (Fig. 2). The clinician-recorded NPPE was moderately correlated with the patient’s pre-visit count of problems to be addressed (r ¼ 0.446; p < 0.001). Visits with female patients comprised 63.5% of the final sample. Patient ages ranged from 18 to 90 years with a median age of 55 years and an average age of 54.6 � 17.5 (sd) years. There were no significant differences in the mean ages among male and female patients. These summary statistics compared favorably with those derived from a large, representative primary care population in Wisconsin covering the same time period (University of Wisconsin Department of Family Medicine Clinical Data Warehouse, 2010; Guil bert et al., 2012). The average age of adult patients, based on 281,982 family practice visits, was 48.8 years; 60.4% of adult primary care pa tients were female. Most clinical encounters were with the patient’s continuity clinician (84.0%). The presenting problems for the clinical encounters were evenly split between acute (43.4%) and chronic (45.1%) medical con ditions. Well care and health maintenance visits comprised an additional 11.5%.
3.4. Determinants of clinician work load in primary care Fig. 5 shows the bivariate relationship between CWL and NPPE. Table 3 shows the relationship that emerged. In order to present more complete information, we present three regression models. The first is the null model of variation between clinicians, the second model pre sents the unadjusted relationship between CWL and NPPE, and the third model looks at the effect of controlling for other variables of interest. We observed a significant relationship between CWL and NPPE. In the un adjusted model, CWL increased by 4.0 (p < 0.001) for each additional problem reported during an encounter. This relationship remained after including clinic, clinician, and additional patient level characteristics in the model (Model 3), with CWL increasing by 3.9 with each additional problem (p < 0.001). Most clinician and patient encounter characteristics did not emerge as significant predictors of CWL. Of those that did, we found that fulltime clinicians had a lower average CWL than their part-time counter parts (31.3 vs. 42.6; p ¼ 0.05). Physician assistants and nurse practi tioners had a similarly reduced average CWL score compared to MDs and DOs (30.1 vs. 42.6; p ¼ 0.02). The CWL for acute presenting problems was only slightly higher than for chronic and other problems (44.2 vs. 42.6; p ¼ 0.20). Other clinician characteristics, including clinician gender, years since training, and years working at the clinic site, did not significantly impact CWL. The results show that CWL varies significantly both within and be tween clinicians and clinics, even after controlling for other effects in the model (p < 0.001 for both estimates, all 3 models). Differences between clinicians in clinics accounted for slightly less of the total variation in the relationship between CWL and NPPE than the within clinic and clinician differences (46% versus 54%; Model 2).
3.3. Clinician workload in primary care The NASA-TLX took 30–45 s to complete following a clinical encounter by clinician report. A wide range of CWL values emerged in usual clinical care, extending from 5.0 to 95.3 points on a 100-point scale (Fig. 3). The mean CWL for all visits was 47.6 � 18.4 (sd) with a median value of 49.3, indicating a moderate workload for routine clinical work (Table 2). Of the domains that make up the workload score, clinicians scored mental demand, time demand, and effort higher on the 100-point scale compared to the other domains. Physical demand had the lowest average score of the six domains (Table 2). The weighted domain scores, based on pairwise ranking within the NASA-TLX, demonstrated that on average, time demands and mental demand contributed 25% each to the final CWL, while physical demand contributed 4%. We observed significant differences in mean CWL score between both the four clinics and 31 clinicians (p < 0.05). Among clinics, mean CWL scores ranged from 39.4 (�18.9) to 56.1 (�15.9). CWL scores exhibited a wider range between clinicians, from a low average score of 18.7 (�10.4) to a high score of 65.8 (�11.5). Among clinicians, there was no association between an individual’s mean NPPE and mean CWL (r ¼
3.5. Outlier analysis Brief interviews with study clinicians regarding encounters with higher-than-expected and lower-than-expected CWL based on NPPE provided valuable insights into unmeasured factors that affect clinical practice. Problem expectation, adjustment of time factors through clinic scheduling, encounter outcome, patient-clinician relationship, and the perceived motivations for the clinical encounter emerged as dominant themes (see Table 4). Factors that are amenable to modification through better human-factors based restructuring of the work environment included assurance that appropriate time is allocated, that the clinician has good continuity of care, and that clear management guidelines are readily available.
Fig. 2. The number of health-related problems evaluated and managed during primary care encounters. Counts were missing for two encounters. 4
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
Fig. 3. Distribution of clinician workload values based for 598 primary care encounters. The NASA-TLX scores were missing for 10 encounters.
In this study, we documented a median CWL of 49.3/100 and assessed the relationship between CWL and the number of problems evaluated and managed during an encounter. The domains of mental demand, time demand, and effort contributed to 71.5% of overall clinician workload. In our sample, clinicians reported evaluating and managing an average of 3.3 problems per encounter, and the CWL increased by about 4 units—or by about 10% of the mean CWL—for each additional problem addressed. There has been little study of primary care CWL in the literature. A study evaluating encounters (one per physician) of 45 family medicine physicians, 20 internists, and 22 neurologists using the NASA-TLX pro vided very similar estimates of mean CWL of 43.4 (95% CI: 40.0–46.8), 40.8 (33.4–48.3), and 48.5 (42. 9–54.1), respectively (Horner et al., 2011). Although the weighted scores were not provided by Horner et al., the unweighted dimensions showed good agreement with our estimates (Horner et al., 2011). In contrast, one study evaluating at internists at Veterans Administration outpatient facilities estimated a lower mean CWL of 31.5 (Rutledge et al., 2009). Interestingly, surgeons appear to experience relatively higher physical demand and lower temporal de mand than did primary care clinicians in our study (Lowndes et al., 2018). Our estimated mean NPPE of 3.30 (range: 1–12), although compa rable, is slightly higher than that found in a previous study in Wisconsin (3.05: range: 1–10) (Beasley et al., 2004), and lower than an estimate from Texas family physicians (3.7; range: 1–10) (Young et al., 2017).
Table 2 Clinician mental workload associated with 598 primary care visits as measured using NASA-TLX. NASA-TLX score
mean
sd
median
Percent of total CWL
47.6
18.4
49.3
100.0
Domain scores
Unweighted Scale Score
Mental demand Physical demand Time demand Performance Effort Frustration
50.4 19.3 51.2 29.7 49.5 34.7
23.9 16.2 24.1 19.4 22.6 24.4
Weighted Scores (%) 50 15 50 25 50 30
24.7 4.0 24.6 13.2 22.2 11.3
4. Discussion Encounters in primary care medicine have been shown to unfold in a chaotic and non-linear manner (Beasley et al., 2011; Holman et al., 2016). It is, therefore, not surprising that primary care clinicians record, on average, a moderate workload. Identifying and further studying factors contributing to this workload may lead to better understanding and mitigating hazards associated with medical error and suboptimal care. At the same time, factors associated with acceptable levels of CWL, especially during otherwise challenging clinical situations, can be used to develop safety-promoting interventions.
Fig. 4. Mean number of problems per encounter and mean clinician workload for each of 31 clinicians. 5
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
Fig. 5. Relationship between clinician workload and number of problems per encounter in primary care practice showing fit of model 3. Table 3 Summary of regression analyses for predictors of clinician workload. Model 1
Model 2
Model 3
Estimate
p value
Estimate
p value
Estimate
p value
47.9
<0.001
34.4
<0.001
42.6
<0.001
4.0
<0.001
3.9
<0.001
Patient encounter characteristics Patient age Continuity patient Nature of presenting problem (acute or not)
0.02 0.66 1.6
0.54 0.70 0.20
Clinician characteristics Clinician schedule (Full time: 1, Part time: 0) Clinician type (PA/NP: 1, MD/DO: 0) Years at clinic site
11.3 12.5 0.17
0.05 0.02 0.52
113.2 157.2
<0.001 <0.001
Intercept Number of problems per encounter
Covariance estimates Between clinician Residual
132.3 209.8
Measure of fit (BIC)
4977
<0.001 <0.001
136.9 161.8 4829
<0.001 <0.001
4209
Notes. Model 1: CWL ¼ intercept (null model), Model 2: CWL ¼ intercept þ NPPE, Model 3: Model 2 þ patient encounter and clinician characteristics.
CWLs reported by the clinicians, indicating some baseline differences in workload amongst clinicians. The number of problems per encounter is a significant feature of primary care medicine and can serve as a proxy for complexity. For example, patient age �65 years and the presence of diabetes were both associated with higher NPPE (Beasley et al., 2004). Despite the hy pothesized effects of competing demands within a limited timeframe, we could not identify any other studies through an extensive literature re view that have evaluated the role of NPPE on CWL. The only other factors emerging from this study as contributing to CWL were full-time status of the clinician, clinician type, and the indi vidual clinician. Full-time clinicians had lower CWL scores. As this study included part-time clinicians within an academic setting, we suspect that competing demands of teaching, research, and administration could increase perceived CWL. Mid-level clinicians, such as physician assis tants and nurse practitioners, reported lower CWL. This is likely due to managing less complex patients. Finally, there was heterogeneity among the clinicians, with a surprisingly wide range of mean CWL values, likely indicating differences in practice composition and coping mechanisms (Rutledge et al., 2009). After adjustment for NPPE, patient age,
Table 4 Emergent themes from outlier analysis of visits with unexpected workload. Lower than Expected Workload ● straightforward problem ● adequate time ● clinician knows patient well ● encounter had good outcome ● patient satisfied ● management clear (standard care plan) ● patient non-demanding
Higher than Expected Workload
● unexpected problems and needs ● being behind, insufficient time ● patient is not known to clinician ● unhappiness or conflict in encounter ● discordant relationship ● unclear decision making, unclear what to do ● demanding, questioning, worked-up, high maintenance, non-responsive patient
Nevertheless, the distribution of NPPE in our study is remarkably similar to that previously reported (Beasley et al., 2004). We were able to document individual differences among clinicians in the mean NPPE recorded. These differences, however, did not translate into the mean 6
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
continuity status, and acuity of presenting problem did not affect CWL. Our evaluation of outliers using a qualitative approach is unique to this study. In doing so, we were able to identify characteristics of en counters that may affect the associated CWL. These characteristics relate to the type of problem (straightforward vs. unexpected; clear vs. nebu lous management), environment (sufficient vs. insufficient time), pa tient (well-known vs. unknown; satisfied vs. conflicted; non-demanding vs. demanding), and relationship (concordant vs. discordant). To the physician authors, these all had high face validity. This provides some guidance for future human factors intervention. For example, processes to better match length of appointment to time demands may be possible based on the moderate correlation between a patient’s pre-visit agenda and the clinician-reported NPPE. Finally, we found that the use of the NASA-TLX was easily incorpo rated into a busy clinical schedule. For this study, we used a paper-based instrument. The availability of the NASA-TLX as a smart phone app or inserted within an electronic medical record system could further facilitate its use. We estimated the additional time needed to complete the NASA TLX following an encounter was less than 45 s.
Finally, the qualitative analysis of 34 encounters occurred one to two months after the encounter allowing for statistical evaluation and outlier identification. Clinicians were provided with a copy of their clinical note. Accordingly, recall bias could have occurred due to the latent period. 4.2. Conclusions Overall, we demonstrated that, on average, ambulatory primary care visits encompassed 3.3 separately identifiable problems and required a moderate level of CWL. CWL was dominated by mental demands, time demands, and overall effort. Significant differences existed among cli nicians in perceived workload. CWL was significantly and positively related to NPPE. Finally, patient, problem, environmental, and rela tionship factors are likely to modify this association. Future human factors-based work is needed to elucidate the relationship between CWL and medical error and possible interventions to better manage CWL, which may include better time allocation, better continuity of care, and more clear management guidelines. (Longo, 2016).
4.1. Limitations
Funding
First and foremost, the data present herein are dated. This study was conducted 10 years ago. The manuscript was complete except for the discussion at the time of Dr. Karsh’s death, at which time additional work ceased. For this special issue, we have attempted to be consistent with his contributions while updating some important points and ref erences. Although dated, we feel that this represents a unique perspec tive on the relationship between CWL and NPPE, which is unlikely to have been affected by the passage of time. As demonstrated by other studies, there has been consistency in the measures of NPPE (Beasley et al., 2004; Young et al., 2017) and CWL (Horner et al., 2011). Second, this study was limited in geospatial distribution. This was a Wisconsin Research and Education Network study for which clinicians were recruited from WREN’s catchment area (Wisconsin Research and Education Network, 2018). Accordingly, regional influences on work load cannot be excluded. Funding for this study limited the number of clinicians that could be evaluated as well. Third, clinical encounters were not randomly selected. We did randomize the time period into one of six possible time slots each day, but medical assistants then chose patients conforming to the study criteria and did not always adhere directly to the time slot. Some se lection bias may have occurred here. This study excluded patients under 18 years of age. Accordingly, measures of NPPE and CWL may have been inflated as younger patients often have fewer co-morbid conditions. Conversely, this study excluded patients with cognitive impairment. Excluding these cases may have lowered CWL. Fourth, the NASA-TLX and the NPPE instruments are somewhat subjective. Although training was provided to clinicians on the di mensions and scales of the NASA-TLX, significant variation in their interpretation and application could occur. Moreover, no firm definition of a clinical problem exists. Rather, we requested clinicians to identify problems that they had “evaluated and managed” during the encounter. Fifth, we were unable to assess the impact of EHR use on CWL. While EHRs were used in two of the clinics at the time of this study, they were in early implementation and clinician placement of orders (e.g., pre scriptions, laboratory tests, X-rays, and referrals) had not yet commenced. The widespread use of the EHR deserves additional study as its use adds distractions (Sinsky and Beasley, 2013) and may create barriers between clinicians and patients (Gawande, 2018) and, there fore, may increase CWL. Sixth, we explored only the CWL during the visit and did not look at additional work occurring outside of direct patient care hours. (Arndt et al., 2017). This additional component of CWL is increasingly recog nized as contributing to overall workload and potentially, clinician burnout (Shanafelt et al., 2016; Sinsky et al., 2016).
This work was supported by the Agency for Healthcare Research and Quality [grant number: 1 R03 HS16026-01] Declaration of competing interest None. Acknowledgements We thank Dr. Michael Grasmick, formerly of WREN, for assistance with recruiting and the participating practices of WREN for research participation. J. Michael Oakes, PhD, Division of Epidemiology and Community Health at the University of Minnesota School of Public Health, consulted on the statistical analyses. Cristalyne Bell assisted with manuscript preparation. References American Academy of Family Physicians. Family Medicine Facts. https://www.aafp. org/about/the-aafp/family -medicine-facts.html. Accessed November 30, 2018. Arndt, B.G., Beasley, J.W., Watkinson, M.D., Temte, J.L., Tuan, W.J., Sinsky, C.A., Gilchrist, V.J., 2017. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann. Fam. Med. 15 (5), 419–426. https://doi.org/10.1370/afm.2121. Baron, R.J., 2010. What’s keeping us to busy in Primary Care? A Snapshot from one practice. NEJM 362, 1632–1636. Beasley, J.W., Hankey, T.H., Erickson, R., Stange, K.C., Mundt, M., Elliott, M., Wiesen, P., Bobula, J., 2004. How many problems do family physicians manage at each encounter? A WREN study. Ann. Fam. Med. 2 (5), 405–410. Beasley, J.W., Wetterneck, T.B., Temte, J., Lapin, J.A., Smith, P., Rivera-Rodriguez, A.J., Karsh, B.T., 2011. Information chaos in primary care: implications for physician performance and patient safety. J. Am. Board Fam. Med. 24, 745–751. https://doi. org/10.3122/jabfm.2011.06.100255. Boultinghouse, O.W., Hammack, G.G., Vo, A.H., Dittmar, M.L., 2007. Assessing physician job satisfaction and mental workload. Telemed. J. e Health 13, 715–718. Byrne, A., 2011. Measurement of mental workload in clinical medicine. Anesth Pain 1, 90–94. Carayon, P. (Ed.), 2006. Handbook of Human Factors and Ergonomics in Patient Safety. Lawrence Erlbaum Associates, Mahwah, NJ. Carayon, P., Gurses, A.P., 2005. A human factors engineering conceptual framework of nursing workload and patient safety in intensive care units. Intensive Crit. Care Nurs. 21 (5), 284–301. Coco, A., 2009. How often do physicians address other medical problems while providing prenatal care? Ann. Fam. Med. 7, 134–138. Primary care: America’s health in a new era. In: Donaldson, M.S., Yordy, K.D., Lohr, K.N., Vanselow, N.A. (Eds.), 1996. Committee on the Future of Primary Care, Division of Health Care Services, Institute of Medicine. National Academy Press, Washington, D. C. Flocke, S.A., Frank, S.H., Wenger, D.A., 2001. Addressing multiple problems in the family practice office visit. J. Fam. Pract. 50 (3), 211–216.
7
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035 Østbye, T., Yarnall, K.S.H., Krause, K.M., Pollak, K.I., Gradison, M., Michener, J.L., 2005. Is there time for management of patients with chronic diseases in primary care? Ann. Fam. Med. 3, 209–214. Rubio, S., Díaz, E., Martín, J., Puente, J., 2004. Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and workload profile methods. Appl. Psychol.: Int. Rev. 53, 61–86. Rutledge, T., Stucky, E., Dollarhide, A., Shively, M., Jain, S., Wolfson, T., Weinger, M.B., Dresselhaus, T., 2009. A real-time assessment of work stress in physicians and nurses. Health Psychol. 28, 194–200. Shanafelt, T.D., Dyrbye, L.N., Sinsky, C., Hasan, O., Satele, D., Sloan, J., West, C., 2016. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin. Proc. 91, 836–848. Sinsky, C.A., Beasley, J.W., 2013. Texting while doctoring: a patient safety hazard. Ann. Intern. Med. 159 (11), 782–783. Sinsky, C., Colligan, L., Li, Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., Blike, G., 2016. Allocation of physician time in ambulatory practice: a time-motion study in four specialties. Ann. Intern. Med. 165, 753–760. Stange, K.C., Zyzanski, S.J., Ja�en, C.R., Callahan, E.J., Kelly, R.B., Gillanders, W.R., Shank, J.C., Chao, J., Medalie, J.H., Miller, W.L., Crabtree, B.F., Flocke, S.A., Gilchrist, V.J., Langa, D.M., Goodwin, M.A., 1998. Illuminating the ‘black box’. A description of 4454 patient visits to 138 family physicians. J. Fam. Pract. 46 (5), 377–389. Temte, J.L., Johnson, T., 2004. Clinicians, practices and patients of the Wisconsin research network [abstract from 2004 Wisconsin primary care research forum]. Wis. Med. J. 103 (7), 75. University of Wisconsin Department of Family Medicine Clinical Data Warehouse. https://cdw.fammed.wisc.edu/. Accessed April 6, 2010. Vidulich, M.A., Tsang, P.S., 2012. Mental workload and situation awareness. In: Salvendy, G. (Ed.), Handbook of Human Factors and Ergonomics, fourth ed. Wiley & Sons, Hoboken, NJ, pp. 243–273. Weinger, M.B., Reddy, S.B., Slage, J.M., 2004. Multiple measures of anesthesia workload during teaching and nonteaching cases. Anesth. Analg. 98, 1419–1425. Wetterneck, T.B., Lapin, J.A., Karsh, B., Beasley, J.W., 2011. Human factors and ergonomics in primary care. In: Carayon, P. (Ed.), Handbook of Human Factors and Ergonomics in Patient Safety, second ed. Taylor and Francis, Boca Raton, FL, pp. 763–775. White, B., 2012. The state of family medicine. Fam. Pract. Manag. 1, 20–26. Wisconsin Research, Education Network. http://www.fammed.wisc.edu/research/wren/ about. Accessed November 30, 2018. Young, G., Zavelina, L., Hooper, V., 2008. Assessment of workload using NASA Task Load Index in perianesthesia nursing. J Perianesth Nurs 23, 102–110. Young, R.A., Burge, S., Kumar, K.A., Wilson, J., 2017. The full scope of family physicians’ work is not reflected by current procedural terminology codes. J. Am. Board Fam. Med. 30, 724–732.
France, D.J., Levin, S., Hemphill, R., Chen, K., Rickard, D., Makowski, R., Jones, I., Aronsky, D., 2005. Emergency physicians’ behaviors and workload in the presence of an electronic whiteboard. Int. J. Med. Inform. 74, 827–837. Gaba, D.M., Lee, T., 1990. Measuring the workload of the anesthesiologist. Anesth. Analg. 71, 354–361. Gawande, A.. Why doctors hate their computers. The New Yorker. https://www.newyork er.com/magazine/2018/11/12/why-doctors-hate-their-computers. December 12, 2018. Guilbert, T.W., Arndt, B., Temte, J., Adams, A., Buckingham, W., Tandias, A., Tomasallo, C., Anderson, H.A., Hanrahan, L.P., 2012. The theory and application of UW eHealth-PHINEX, a clinical electronic health record–public health information exchange. Wis. Med. J. 111 (3), 124–133. Hart, S., Staveland, L., 1988. Development of NASA-TLX (task Load Index): results of empirical and theoretical research. In: Hancock, P., Meshkati, N. (Eds.), Human Mental Workload. North Holland, Amsterdam, pp. 139–183. Helmreich, R.L., 2000. On error management: lessons from aviation. BMJ 320, 781–785. Holden, R.J., Patel, N.R., Scanlon, M.C., Shalaby, T.M., Arnold, J.M., Karsh, B., 2010. Effects of mental demands during dispensing on perceived medication safety and employee well being: a study of workload in pediatric hospital pharmacies. Res. Soc. Adm. Pharm. 6 (4), 293–306. Holden, R.J., Scanlon, M.C., Patel, N.R., Kaushal, R., Escoto, K.H., Brown, R.L., Alper, S. J., Arnold, J.M., Shalaby, T.M., Murkowski, K., Karsh, B.T., 2011. A human factors framework and study of the effect of nursing workload on patient safety and employee quality of working life. BMJ Qual. Saf. 20 (1), 15–24. Holman, G.T., Beasley, J.W., Karsh, B.-T., Stone, J.A., Smith, P.D., Wetterneck, T.B., 2016. The myth of standardized workflow in primary care. J. Am. Med. Inform. Assoc. 23, 29–37. Horner, R.D., Szaflarski, J.P., Ying, J., Meganathan, K., Matthews, G., Schroer, B., Weber, D., Raphaelson, M., 2011. Physician work intensity among medical specialties: emerging evidence on its magnitude and composition. Med. Care 49, 1007–1011. Karsh, B., Holden, R.J., Alper, S.J., Or, K.L., 2006. A human factors engineering paradigm for patient safety: designing to support the performance of the health care professional. Qual. Saf. Health Care 15 (Suppl. I), i59–i65. Katerndahl, D.A., Wood, R., Ja�en, C.R., 2010. A method for estimating relative complexity of ambulatory care. Ann. Fam. Med. 8, 341–347. Leedal, J.M., Smith, A.F., 2005. Methodological approaches to anaesthetists’ workload in the operating theatre. Br. J. Anaesth. 94, 702–709. Longo, L., 2016. Mental workload in medicine: foundations, applications, open problems, challenges and future perspectives. Computer-based medical systems (CBMS) 29th. IEEE International Symposium 30, 24. June. Lowndes, B.R., Forsyth, K.L., Blocker, R.C., Dean, P.G., Truty, M.J., Heller, S.F., Blackmon, S., Hallbeck, M.S., Nelson, H., 2018 Sep 20. NASA-TLX assessment of surgeon workload variation across specialties. Ann. Surg. https://doi.org/10.1097/ SLA.0000000000003058 ([Epub ahead of print]).
8
Contents lists available at ScienceDirect
Applied Ergonomics journal homepage: http://www.elsevier.com/locate/apergo
Relationship between number of health problems addressed during a primary care patient visit and clinician workload Jonathan L. Temte a, *, John W. Beasley a, c, Richard J. Holden b, Ben-Tzion Karsh c, 1, Beth Potter a, Paul Smith a, Peggy O’Halloran d a
University of Wisconsin School of Medicine and Public Health, Department of Family Medicine and Community Health, 1100 Delaplaine Court, Madison, WI, 53715, USA Indiana University School of Medicine, Department of Medicine 545 Barnhill Dr., Emerson Hall 305, Indianapolis, IN, 46202, USA c University of Wisconsin, Department of Industrial and Systems Engineering, 1415 Engineering Drive, Madison, WI, 53706, USA d Eau Claire City-County Health Department, 720 2nd Ave, Eau Claire, WI, 54703, USA b
A R T I C L E I N F O
A B S T R A C T
Introduction: Primary care is complex due to multiple health problems being addressed in each patient visit. Little is known about the effect of the number of problems per encounter (NPPE) on the resulting clinician workload (CWL), as measured using the National Aeronautics and Space Administration Task Load Index (NASA-TLX). Methods: We evaluated the relationship between NPPE and CWL across 608 adult patient visits, conducted by 31 clinicians, using hierarchical linear regression. Clinicians were interviewed about outlier visits to identify reasons for higher or lower than expected CWL. Results: Mean NPPE was 3.30 � 2.0 (sd) and CWL was 47.6 � 18.4 from a maximum of 100. Mental demand, time demand and effort accounted for 71.5% of CWL. After adjustment for confounders, each additional problem increased CWL by 3.9 points (P < 0.001). Patient, problem, environmental and patient-physician relationship factors were qualitatively identified from interviews as moderators of this effect. Conclusion: CWL is positively related to NPPE. Several modifiable factors may enhance or mitigate this effect. Our findings have implications for using a Human Factors (HF) approach to managing CWL.
1. Introduction The provision of primary medical care is a highly complex process (Wetterneck et al., 2011; Beasley et al., 2011; Holman et al., 2016) for which measurements of complexity have been proposed (Katerndahl et al., 2010). As compared to sub-specialty care, in which a single health problem within a single organ system is typically addressed, primary care spans the spectrum from disease prevention to end-stage disease and palliative care, from acute care to chronic disease management, and across all organ systems, age ranges and genders (Donaldson et al., 1996). Consequently, multiple and often unrelated and competing problems are bundled together within the context of a single visit (Beasley et al., 2004; Stange et al., 1998; Flocke et al., 2001; Coco,
2009). Across 34 h of direct patient care (White, 2012), the average family physician provides 84 outpatient visits each week (American Academy of Family Physicians, 2018), with each visit involving, on average, the evaluation and management of three or more problems (Beasley et al., 2004), thus resulting in more than 256 distinct problems per week. A problem is defined as a symptom, disease, abnormality (e.g. x-ray), psychological or social issue about which the clinician gathers data and makes a decision. Problems can include an acute health-related issue (e. g., sore throat), a chronic disease management issue (e.g., control of hypertension), a wellness issue (e.g., arranging a colonoscopy), or a psychosocial issue (e.g., concern about a spouse’s alcohol consumption), among others. This estimate does not take into account the additional
Abbreviations: CWL, clinician workload; EHR, electronic health record; NASA-TLX, National Aeronautics and Space Administration Task Load Index; NPPE, number of problems per encounter; WREN, Wisconsin Research and Education Network. * Corresponding author. Family Medicine and Community Health University of Wisconsin School of Medicine and Public Health, 1100 Delaplaine Court, Madison, WI, 53715, USA. E-mail addresses: [email protected] (J.L. Temte), [email protected] (B.-T. Karsh). 1 Deceased. https://doi.org/10.1016/j.apergo.2019.103035 Received 14 December 2018; Received in revised form 9 July 2019; Accepted 13 December 2019 0003-6870/© 2019 Elsevier Ltd. All rights reserved.
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
2018) practices that were selected to form a diverse and representative set of primary care sites, which included two urban and two rural set tings. After obtaining in-person informed consent, we trained clinicians to record data on routine adult patient visits over a four-week study period. Each clinician was instructed to provide data for 20 patient visits in order to achieve the initial target sample size of 600 clinical en counters. Data were collected between August 2007 and March 2008; whereas electronic health records (EHR) had been implemented at two sites, order entry using EHR—a time consuming task for the clin ician—had not yet commenced at this time. Clinic days were divided into early, mid, and late morning sessions and early, mid, and late afternoon sessions. On any given day, the cli nician’s medical assistant selected a patient from a randomly assigned time period. This quasi-randomization process was used to provide a representative patient population. No incentives were used for recruit ment. The only stipulations for recruitment were that the patient be an adult (age 18 or greater), not be a prisoner or institutionalized, and be free of significant impairment in decision-making capacity. No other inclusion/exclusion criteria were used so as to mirror the types of pa tients typically seen by primary care clinicians. The study protocol was approved by the University of Wisconsin Health Sciences Institutional Review Board. Following patient recruitment and obtaining informed consent by a trained clinical staff member, a medical assistant or nurse collected basic demographic data. At the time of rooming and before any contact with the study clinicians, patients identified the number of problems that they wanted to have addressed during the forthcoming encounter. Following the visit, the patient recorded the extent to which the clinician addressed their concerns using a 7-point Likert scale. The clinician did not have access to this information at any time. At the conclusion of the patient visit, the clinician completed three instruments to characterize the clinical encounter. The number of problems per encounter (NPPE) was recorded by the clinician who identified both the number and types of health related problems addressed during the patient visit using a form modified from the study of Beasley et al. (2004) A copy of the corresponding clinical encounter note was de-identified, labeled with the clinician’s ID and the encounter number, and attached with the study forms. This was available for reference if the clinician recorded problem log required clarification. CWL for the encounter was measured immediately following the visit using the NASA-TLX (Hart et al., 1988). The NASA-TLX is a weighted combination of six dimensions of work (mental demand, physical de mand, time demand, performance, effort and frustration) that reliably
burden imposed by inpatient care, home and nursing home visits, and management of problems by telephone, electronic communication and through intermediary clinical staff (Baron, 2010; Arndt et al., 2017). There has been comment on the impossible task of fully adhering to chronic disease guidelines within this severely limited timeframe (Østbye et al., 2005). Moreover, there has been recognition that critical decisions about patient care require high mental acuity (Helmreich, 2000; Young et al., 2008). Little attention, however, has been paid to the additive effect of managing multiple problems on clinician workload (CWL). Given the relationship between CWL and the likelihood of errors (Vidulich et al., 2012), a positive linear association between the number of problems and CWL would imply each problem a clinician evaluates and manages during a patient visit might increase the potential for medical error. We hypothesized that a relationship exists linking the number of problems managed during a single clinical visit and the resultant CWL for that visit. Furthermore, we hypothesized, but did not test in this study, that CWL is related to visit outcomes such as quality of care, medical error, and clinician burnout. The overall conceptual framework is illustrated in Fig. 1 and draws from prior models and empirical studies of CWL in healthcare (Holden et al., 2010, 2011; Carayon and Gurses, 2005). In this study, we used a validated instrument to measure the CWL of a diverse set of primary care clinicians within four practices affiliated with an established practice-based research network, the Wisconsin Research and Education Network (WREN) (Temte and Johnson, 2004; Wisconsin Research and Education Network, 2018). We examined whether one component of complexity—an increase in the number of problems a clinician addresses during a single patient visit—was associated with an increase in CWL as measured by the National Aeronautics and Space Administration-Task Load Index (NASA-TLX) (Hart et al., 1988; Byrne, 2011). We also assessed the relationship between CWL and other clinic, clinician, and patient visit characteristics. Finally, we employed a qualitative analysis of outlying observations to collect additional infor mation about the relationship between the number of problems addressed during a patient visit and resulting CWL. 2. Methods 2.1. Data We recruited thirty-one primary care clinicians from four WREN (Temte and Johnson, 2004; Wisconsin Research and Education Network,
Fig. 1. Simplified model of clinician workload in primary care, which may serve as a conceptual basis for future evaluations. This study tested components within the grey box. 2
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
measures workload (Hart et al., 1988; Byrne, 2011; Rubio et al., 2004). Moreover, this is a validated and commonly used instrument in human factors engineering (Karsh et al., 2006; Carayon, 2006). Although the NASA-TLX has had some use in health care settings, including emer gency medicine (France et al., 2005), telemedicine (Boultinghouse et al., 2007) and anesthesia (Byrne, 2011; Leedal and Smith, 2005; Weinger et al., 2004; Gaba and Lee, 1990), it had not been previously utilized in primary care settings. Finally, using three separate 7-point Likert scales, clinicians completed a form that assessed the extent to which they felt they addressed patient concerns and the estimated likelihood of possible omission and/or error (data for visit completeness, omissions, and errors were not used in this current assessment). In pilot tests, the total time required for all clinician instrument completion was less than 2 min. We purposely collected data for no more than one encounter per half day so as not to contribute additionally to clinician workload.
3. Results 3.1. Clinic and clinician characteristics Thirty-one clinicians from four WREN-affiliated primary care clinics participated in this study. Table 1 shows the characteristics of the participating clinics and clinicians, and describes the study patient en counters. Three of the clinics were located in Wisconsin; the fourth was located in Northeast Iowa on the Wisconsin border. Two of the sites were urban (17 clinicians) and two were rural (14 clinicians). Both urban practices were actively utilizing electronic medical record systems. Physicians, physician assistants, and nurse practitioners participated in the study, representing family medicine and internal medicine practices. Most participants were engaged in full-time practice and the clinicians tended to be experienced. On average, 15.5 years had elapsed since completion of training (median ¼ 14 years; range: 1–32 years) and the clinicians had been practicing at their current clinical sites for an average of 12.9 years (median ¼ 13.5 years; range: 1–28 years).
2.2. Analysis
3.2. Patient encounters
We initially examined associations between CWL and NPPE and other potential predictors of CWL using Pearson correlation coefficients and plotted the bivariate relationship between NPPE and CWL. We then used a hierarchical linear regression model (SAS 9.1) to assess the relationship between NPPE and CWL, accounting for the correlation of patients within clinicians at each of the four participating clinics. We accounted for the correlation of patients within clinician, and clinician within clinic by including clinicians within clinics as a random effect in the model. As is commonly observed, a single random effect in the final models accounted for the intra-group correlation. Finally, we included clinic, clinician, and patient characteristics in the model to identify other predictors of CWL and adjust the NPPE-CWL association. We excluded some potential confounders (practice: rural vs. urban; type and EHR use; patient: gender, day and time of appointment; clinician: gender, years out of training) from the multivariate analysis due to issues of collin earity or a lack of a significant correlation with CWL. We used Bayesian information criterion (BIC) to allow for model selection among a set of models, wherein the lowest BIC was preferred. BIC approaches over fitting of a model by introducing a penalty term for the number of pa rameters in the model.
We collected data from 608 patient encounters conducted by 31 study clinicians (mean ¼ 19.6 encounters per clinician; median ¼ 20) from August 2007 through March 2008 (Table 1). Complete data were available for 598 (98.4%) encounters. Study visits occurred more commonly on Tuesdays than expected and less commonly on Fridays than expected (X2 ¼ 10.3; df ¼ 4; P ¼ 0.03). Early afternoon visits were under-represented and mid-afternoon visits were over-represented in the final sample (X2 ¼ 18.0; df ¼ 5; P ¼ 0.003). Overall, however, there was good representation of encounters—from 8 to 29 visits—in each of 30 potential time slots. Table 1 Practice, clinician, and patient encounter characteristics. Practice Characteristics (n ¼ 4) Setting
Rural Urban Practice Type Family Medicine Internal and Family Medicine EHR in Use Yes No Clinician Characteristics (n ¼ 31) Gender Male Female Degree MD/DO PA/NP Schedule Part time Full time Years since completing training Years at practice site Patient Encounter Characteristics (n ¼ 608) Number of Problems 1–12 Patient Age 18–90 years
2.3. Qualitative assessment To assess important qualitative characteristics, we conducted brief clinician interviews on a subset of visits (n ¼ 34; 5.6% of all study visits) that had unexpectedly high or low CWLs relative to the number of problems. Eighteen of these events were selected based on an absolute standardized residual of 2.0 or greater from the clinician-specific simple linear regression model relating CWL to NPPE. Sixteen additional included events did not meet the above statistical definition of an outlier but were notably displaced from the regression line in graphical plots of CWL vs. NPPE. For these 34 total encounters, the de-identified clinical note was faxed to the clinician for review, and a brief interview was conducted in person or by phone. Interviews were conducted only if the visit was clearly remembered by the clinician (all visits were remem bered). A list of open-ended questions was presented to the clinician pertaining to anything that stood out about the visit, factors that may have contributed to a greater or lesser CWL, the relationship between the clinician and the patient, and contributing factors of clinic setting or schedule. The interviewers kept detailed notes regarding the salient points of the interview. The notes were typed and provided to three clinically active family physician reviewers (JB, BP, JT) for assessment and identification of themes associated with high and low CWL. The final list of factors was constructed following joint review of each case and development of consensus.
Patient Gender Continuity Patient Status Day of Appointment
Time of Appointment Nature of Presenting Problem
3
Male Female Yes No Monday Tuesday Wednesday Thursday Friday Morning Afternoon Acute Problem Chronic Problem Well Care Other
2 2 3 1 2 2 19 (61.3%) 12 (38.7%) 24 (77.4%) 7 (22.6%) 5 (16.1%) 26 (83.9%) 15.5 � 8.2 (sd) years 12.9 � 8.2 (sd) years 3.3 � 2.0 (sd) 54.6 � 17.5 (sd) years 221 (36.5%) 384 (63.5%) 450 (84.0%) 86 (16.0%) 123 (21.1%) 136 (23.3%) 114 (19.6%) 121 (20.8%) 89 (15.3%) 275 (52.7%) 247 (47.3%) 255 (43.4%) 265 (45.1%) 58 (9.9%) 10 (1.6%)
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
0.217; p ¼ 0.240; Fig. 4).
At the start of the encounter, patients reported an average of 2.32 � 1.42 (sd) problems they wished to have addressed. After the visit, they overwhelmingly reported that all their problems were addressed during the visit (91% responded at the highest level on the 7-point Likert scale). Clinicians reported evaluating and managing from 1 to 12 problems during each encounter, with an average NPPE of 3.30 � 2.0 (sd) and a median of 3 problems (Fig. 2). The clinician-recorded NPPE was moderately correlated with the patient’s pre-visit count of problems to be addressed (r ¼ 0.446; p < 0.001). Visits with female patients comprised 63.5% of the final sample. Patient ages ranged from 18 to 90 years with a median age of 55 years and an average age of 54.6 � 17.5 (sd) years. There were no significant differences in the mean ages among male and female patients. These summary statistics compared favorably with those derived from a large, representative primary care population in Wisconsin covering the same time period (University of Wisconsin Department of Family Medicine Clinical Data Warehouse, 2010; Guil bert et al., 2012). The average age of adult patients, based on 281,982 family practice visits, was 48.8 years; 60.4% of adult primary care pa tients were female. Most clinical encounters were with the patient’s continuity clinician (84.0%). The presenting problems for the clinical encounters were evenly split between acute (43.4%) and chronic (45.1%) medical con ditions. Well care and health maintenance visits comprised an additional 11.5%.
3.4. Determinants of clinician work load in primary care Fig. 5 shows the bivariate relationship between CWL and NPPE. Table 3 shows the relationship that emerged. In order to present more complete information, we present three regression models. The first is the null model of variation between clinicians, the second model pre sents the unadjusted relationship between CWL and NPPE, and the third model looks at the effect of controlling for other variables of interest. We observed a significant relationship between CWL and NPPE. In the un adjusted model, CWL increased by 4.0 (p < 0.001) for each additional problem reported during an encounter. This relationship remained after including clinic, clinician, and additional patient level characteristics in the model (Model 3), with CWL increasing by 3.9 with each additional problem (p < 0.001). Most clinician and patient encounter characteristics did not emerge as significant predictors of CWL. Of those that did, we found that fulltime clinicians had a lower average CWL than their part-time counter parts (31.3 vs. 42.6; p ¼ 0.05). Physician assistants and nurse practi tioners had a similarly reduced average CWL score compared to MDs and DOs (30.1 vs. 42.6; p ¼ 0.02). The CWL for acute presenting problems was only slightly higher than for chronic and other problems (44.2 vs. 42.6; p ¼ 0.20). Other clinician characteristics, including clinician gender, years since training, and years working at the clinic site, did not significantly impact CWL. The results show that CWL varies significantly both within and be tween clinicians and clinics, even after controlling for other effects in the model (p < 0.001 for both estimates, all 3 models). Differences between clinicians in clinics accounted for slightly less of the total variation in the relationship between CWL and NPPE than the within clinic and clinician differences (46% versus 54%; Model 2).
3.3. Clinician workload in primary care The NASA-TLX took 30–45 s to complete following a clinical encounter by clinician report. A wide range of CWL values emerged in usual clinical care, extending from 5.0 to 95.3 points on a 100-point scale (Fig. 3). The mean CWL for all visits was 47.6 � 18.4 (sd) with a median value of 49.3, indicating a moderate workload for routine clinical work (Table 2). Of the domains that make up the workload score, clinicians scored mental demand, time demand, and effort higher on the 100-point scale compared to the other domains. Physical demand had the lowest average score of the six domains (Table 2). The weighted domain scores, based on pairwise ranking within the NASA-TLX, demonstrated that on average, time demands and mental demand contributed 25% each to the final CWL, while physical demand contributed 4%. We observed significant differences in mean CWL score between both the four clinics and 31 clinicians (p < 0.05). Among clinics, mean CWL scores ranged from 39.4 (�18.9) to 56.1 (�15.9). CWL scores exhibited a wider range between clinicians, from a low average score of 18.7 (�10.4) to a high score of 65.8 (�11.5). Among clinicians, there was no association between an individual’s mean NPPE and mean CWL (r ¼
3.5. Outlier analysis Brief interviews with study clinicians regarding encounters with higher-than-expected and lower-than-expected CWL based on NPPE provided valuable insights into unmeasured factors that affect clinical practice. Problem expectation, adjustment of time factors through clinic scheduling, encounter outcome, patient-clinician relationship, and the perceived motivations for the clinical encounter emerged as dominant themes (see Table 4). Factors that are amenable to modification through better human-factors based restructuring of the work environment included assurance that appropriate time is allocated, that the clinician has good continuity of care, and that clear management guidelines are readily available.
Fig. 2. The number of health-related problems evaluated and managed during primary care encounters. Counts were missing for two encounters. 4
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
Fig. 3. Distribution of clinician workload values based for 598 primary care encounters. The NASA-TLX scores were missing for 10 encounters.
In this study, we documented a median CWL of 49.3/100 and assessed the relationship between CWL and the number of problems evaluated and managed during an encounter. The domains of mental demand, time demand, and effort contributed to 71.5% of overall clinician workload. In our sample, clinicians reported evaluating and managing an average of 3.3 problems per encounter, and the CWL increased by about 4 units—or by about 10% of the mean CWL—for each additional problem addressed. There has been little study of primary care CWL in the literature. A study evaluating encounters (one per physician) of 45 family medicine physicians, 20 internists, and 22 neurologists using the NASA-TLX pro vided very similar estimates of mean CWL of 43.4 (95% CI: 40.0–46.8), 40.8 (33.4–48.3), and 48.5 (42. 9–54.1), respectively (Horner et al., 2011). Although the weighted scores were not provided by Horner et al., the unweighted dimensions showed good agreement with our estimates (Horner et al., 2011). In contrast, one study evaluating at internists at Veterans Administration outpatient facilities estimated a lower mean CWL of 31.5 (Rutledge et al., 2009). Interestingly, surgeons appear to experience relatively higher physical demand and lower temporal de mand than did primary care clinicians in our study (Lowndes et al., 2018). Our estimated mean NPPE of 3.30 (range: 1–12), although compa rable, is slightly higher than that found in a previous study in Wisconsin (3.05: range: 1–10) (Beasley et al., 2004), and lower than an estimate from Texas family physicians (3.7; range: 1–10) (Young et al., 2017).
Table 2 Clinician mental workload associated with 598 primary care visits as measured using NASA-TLX. NASA-TLX score
mean
sd
median
Percent of total CWL
47.6
18.4
49.3
100.0
Domain scores
Unweighted Scale Score
Mental demand Physical demand Time demand Performance Effort Frustration
50.4 19.3 51.2 29.7 49.5 34.7
23.9 16.2 24.1 19.4 22.6 24.4
Weighted Scores (%) 50 15 50 25 50 30
24.7 4.0 24.6 13.2 22.2 11.3
4. Discussion Encounters in primary care medicine have been shown to unfold in a chaotic and non-linear manner (Beasley et al., 2011; Holman et al., 2016). It is, therefore, not surprising that primary care clinicians record, on average, a moderate workload. Identifying and further studying factors contributing to this workload may lead to better understanding and mitigating hazards associated with medical error and suboptimal care. At the same time, factors associated with acceptable levels of CWL, especially during otherwise challenging clinical situations, can be used to develop safety-promoting interventions.
Fig. 4. Mean number of problems per encounter and mean clinician workload for each of 31 clinicians. 5
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
Fig. 5. Relationship between clinician workload and number of problems per encounter in primary care practice showing fit of model 3. Table 3 Summary of regression analyses for predictors of clinician workload. Model 1
Model 2
Model 3
Estimate
p value
Estimate
p value
Estimate
p value
47.9
<0.001
34.4
<0.001
42.6
<0.001
4.0
<0.001
3.9
<0.001
Patient encounter characteristics Patient age Continuity patient Nature of presenting problem (acute or not)
0.02 0.66 1.6
0.54 0.70 0.20
Clinician characteristics Clinician schedule (Full time: 1, Part time: 0) Clinician type (PA/NP: 1, MD/DO: 0) Years at clinic site
11.3 12.5 0.17
0.05 0.02 0.52
113.2 157.2
<0.001 <0.001
Intercept Number of problems per encounter
Covariance estimates Between clinician Residual
132.3 209.8
Measure of fit (BIC)
4977
<0.001 <0.001
136.9 161.8 4829
<0.001 <0.001
4209
Notes. Model 1: CWL ¼ intercept (null model), Model 2: CWL ¼ intercept þ NPPE, Model 3: Model 2 þ patient encounter and clinician characteristics.
CWLs reported by the clinicians, indicating some baseline differences in workload amongst clinicians. The number of problems per encounter is a significant feature of primary care medicine and can serve as a proxy for complexity. For example, patient age �65 years and the presence of diabetes were both associated with higher NPPE (Beasley et al., 2004). Despite the hy pothesized effects of competing demands within a limited timeframe, we could not identify any other studies through an extensive literature re view that have evaluated the role of NPPE on CWL. The only other factors emerging from this study as contributing to CWL were full-time status of the clinician, clinician type, and the indi vidual clinician. Full-time clinicians had lower CWL scores. As this study included part-time clinicians within an academic setting, we suspect that competing demands of teaching, research, and administration could increase perceived CWL. Mid-level clinicians, such as physician assis tants and nurse practitioners, reported lower CWL. This is likely due to managing less complex patients. Finally, there was heterogeneity among the clinicians, with a surprisingly wide range of mean CWL values, likely indicating differences in practice composition and coping mechanisms (Rutledge et al., 2009). After adjustment for NPPE, patient age,
Table 4 Emergent themes from outlier analysis of visits with unexpected workload. Lower than Expected Workload ● straightforward problem ● adequate time ● clinician knows patient well ● encounter had good outcome ● patient satisfied ● management clear (standard care plan) ● patient non-demanding
Higher than Expected Workload
● unexpected problems and needs ● being behind, insufficient time ● patient is not known to clinician ● unhappiness or conflict in encounter ● discordant relationship ● unclear decision making, unclear what to do ● demanding, questioning, worked-up, high maintenance, non-responsive patient
Nevertheless, the distribution of NPPE in our study is remarkably similar to that previously reported (Beasley et al., 2004). We were able to document individual differences among clinicians in the mean NPPE recorded. These differences, however, did not translate into the mean 6
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035
continuity status, and acuity of presenting problem did not affect CWL. Our evaluation of outliers using a qualitative approach is unique to this study. In doing so, we were able to identify characteristics of en counters that may affect the associated CWL. These characteristics relate to the type of problem (straightforward vs. unexpected; clear vs. nebu lous management), environment (sufficient vs. insufficient time), pa tient (well-known vs. unknown; satisfied vs. conflicted; non-demanding vs. demanding), and relationship (concordant vs. discordant). To the physician authors, these all had high face validity. This provides some guidance for future human factors intervention. For example, processes to better match length of appointment to time demands may be possible based on the moderate correlation between a patient’s pre-visit agenda and the clinician-reported NPPE. Finally, we found that the use of the NASA-TLX was easily incorpo rated into a busy clinical schedule. For this study, we used a paper-based instrument. The availability of the NASA-TLX as a smart phone app or inserted within an electronic medical record system could further facilitate its use. We estimated the additional time needed to complete the NASA TLX following an encounter was less than 45 s.
Finally, the qualitative analysis of 34 encounters occurred one to two months after the encounter allowing for statistical evaluation and outlier identification. Clinicians were provided with a copy of their clinical note. Accordingly, recall bias could have occurred due to the latent period. 4.2. Conclusions Overall, we demonstrated that, on average, ambulatory primary care visits encompassed 3.3 separately identifiable problems and required a moderate level of CWL. CWL was dominated by mental demands, time demands, and overall effort. Significant differences existed among cli nicians in perceived workload. CWL was significantly and positively related to NPPE. Finally, patient, problem, environmental, and rela tionship factors are likely to modify this association. Future human factors-based work is needed to elucidate the relationship between CWL and medical error and possible interventions to better manage CWL, which may include better time allocation, better continuity of care, and more clear management guidelines. (Longo, 2016).
4.1. Limitations
Funding
First and foremost, the data present herein are dated. This study was conducted 10 years ago. The manuscript was complete except for the discussion at the time of Dr. Karsh’s death, at which time additional work ceased. For this special issue, we have attempted to be consistent with his contributions while updating some important points and ref erences. Although dated, we feel that this represents a unique perspec tive on the relationship between CWL and NPPE, which is unlikely to have been affected by the passage of time. As demonstrated by other studies, there has been consistency in the measures of NPPE (Beasley et al., 2004; Young et al., 2017) and CWL (Horner et al., 2011). Second, this study was limited in geospatial distribution. This was a Wisconsin Research and Education Network study for which clinicians were recruited from WREN’s catchment area (Wisconsin Research and Education Network, 2018). Accordingly, regional influences on work load cannot be excluded. Funding for this study limited the number of clinicians that could be evaluated as well. Third, clinical encounters were not randomly selected. We did randomize the time period into one of six possible time slots each day, but medical assistants then chose patients conforming to the study criteria and did not always adhere directly to the time slot. Some se lection bias may have occurred here. This study excluded patients under 18 years of age. Accordingly, measures of NPPE and CWL may have been inflated as younger patients often have fewer co-morbid conditions. Conversely, this study excluded patients with cognitive impairment. Excluding these cases may have lowered CWL. Fourth, the NASA-TLX and the NPPE instruments are somewhat subjective. Although training was provided to clinicians on the di mensions and scales of the NASA-TLX, significant variation in their interpretation and application could occur. Moreover, no firm definition of a clinical problem exists. Rather, we requested clinicians to identify problems that they had “evaluated and managed” during the encounter. Fifth, we were unable to assess the impact of EHR use on CWL. While EHRs were used in two of the clinics at the time of this study, they were in early implementation and clinician placement of orders (e.g., pre scriptions, laboratory tests, X-rays, and referrals) had not yet commenced. The widespread use of the EHR deserves additional study as its use adds distractions (Sinsky and Beasley, 2013) and may create barriers between clinicians and patients (Gawande, 2018) and, there fore, may increase CWL. Sixth, we explored only the CWL during the visit and did not look at additional work occurring outside of direct patient care hours. (Arndt et al., 2017). This additional component of CWL is increasingly recog nized as contributing to overall workload and potentially, clinician burnout (Shanafelt et al., 2016; Sinsky et al., 2016).
This work was supported by the Agency for Healthcare Research and Quality [grant number: 1 R03 HS16026-01] Declaration of competing interest None. Acknowledgements We thank Dr. Michael Grasmick, formerly of WREN, for assistance with recruiting and the participating practices of WREN for research participation. J. Michael Oakes, PhD, Division of Epidemiology and Community Health at the University of Minnesota School of Public Health, consulted on the statistical analyses. Cristalyne Bell assisted with manuscript preparation. References American Academy of Family Physicians. Family Medicine Facts. https://www.aafp. org/about/the-aafp/family -medicine-facts.html. Accessed November 30, 2018. Arndt, B.G., Beasley, J.W., Watkinson, M.D., Temte, J.L., Tuan, W.J., Sinsky, C.A., Gilchrist, V.J., 2017. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann. Fam. Med. 15 (5), 419–426. https://doi.org/10.1370/afm.2121. Baron, R.J., 2010. What’s keeping us to busy in Primary Care? A Snapshot from one practice. NEJM 362, 1632–1636. Beasley, J.W., Hankey, T.H., Erickson, R., Stange, K.C., Mundt, M., Elliott, M., Wiesen, P., Bobula, J., 2004. How many problems do family physicians manage at each encounter? A WREN study. Ann. Fam. Med. 2 (5), 405–410. Beasley, J.W., Wetterneck, T.B., Temte, J., Lapin, J.A., Smith, P., Rivera-Rodriguez, A.J., Karsh, B.T., 2011. Information chaos in primary care: implications for physician performance and patient safety. J. Am. Board Fam. Med. 24, 745–751. https://doi. org/10.3122/jabfm.2011.06.100255. Boultinghouse, O.W., Hammack, G.G., Vo, A.H., Dittmar, M.L., 2007. Assessing physician job satisfaction and mental workload. Telemed. J. e Health 13, 715–718. Byrne, A., 2011. Measurement of mental workload in clinical medicine. Anesth Pain 1, 90–94. Carayon, P. (Ed.), 2006. Handbook of Human Factors and Ergonomics in Patient Safety. Lawrence Erlbaum Associates, Mahwah, NJ. Carayon, P., Gurses, A.P., 2005. A human factors engineering conceptual framework of nursing workload and patient safety in intensive care units. Intensive Crit. Care Nurs. 21 (5), 284–301. Coco, A., 2009. How often do physicians address other medical problems while providing prenatal care? Ann. Fam. Med. 7, 134–138. Primary care: America’s health in a new era. In: Donaldson, M.S., Yordy, K.D., Lohr, K.N., Vanselow, N.A. (Eds.), 1996. Committee on the Future of Primary Care, Division of Health Care Services, Institute of Medicine. National Academy Press, Washington, D. C. Flocke, S.A., Frank, S.H., Wenger, D.A., 2001. Addressing multiple problems in the family practice office visit. J. Fam. Pract. 50 (3), 211–216.
7
J.L. Temte et al.
Applied Ergonomics 84 (2020) 103035 Østbye, T., Yarnall, K.S.H., Krause, K.M., Pollak, K.I., Gradison, M., Michener, J.L., 2005. Is there time for management of patients with chronic diseases in primary care? Ann. Fam. Med. 3, 209–214. Rubio, S., Díaz, E., Martín, J., Puente, J., 2004. Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and workload profile methods. Appl. Psychol.: Int. Rev. 53, 61–86. Rutledge, T., Stucky, E., Dollarhide, A., Shively, M., Jain, S., Wolfson, T., Weinger, M.B., Dresselhaus, T., 2009. A real-time assessment of work stress in physicians and nurses. Health Psychol. 28, 194–200. Shanafelt, T.D., Dyrbye, L.N., Sinsky, C., Hasan, O., Satele, D., Sloan, J., West, C., 2016. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction. Mayo Clin. Proc. 91, 836–848. Sinsky, C.A., Beasley, J.W., 2013. Texting while doctoring: a patient safety hazard. Ann. Intern. Med. 159 (11), 782–783. Sinsky, C., Colligan, L., Li, Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., Blike, G., 2016. Allocation of physician time in ambulatory practice: a time-motion study in four specialties. Ann. Intern. Med. 165, 753–760. Stange, K.C., Zyzanski, S.J., Ja�en, C.R., Callahan, E.J., Kelly, R.B., Gillanders, W.R., Shank, J.C., Chao, J., Medalie, J.H., Miller, W.L., Crabtree, B.F., Flocke, S.A., Gilchrist, V.J., Langa, D.M., Goodwin, M.A., 1998. Illuminating the ‘black box’. A description of 4454 patient visits to 138 family physicians. J. Fam. Pract. 46 (5), 377–389. Temte, J.L., Johnson, T., 2004. Clinicians, practices and patients of the Wisconsin research network [abstract from 2004 Wisconsin primary care research forum]. Wis. Med. J. 103 (7), 75. University of Wisconsin Department of Family Medicine Clinical Data Warehouse. https://cdw.fammed.wisc.edu/. Accessed April 6, 2010. Vidulich, M.A., Tsang, P.S., 2012. Mental workload and situation awareness. In: Salvendy, G. (Ed.), Handbook of Human Factors and Ergonomics, fourth ed. Wiley & Sons, Hoboken, NJ, pp. 243–273. Weinger, M.B., Reddy, S.B., Slage, J.M., 2004. Multiple measures of anesthesia workload during teaching and nonteaching cases. Anesth. Analg. 98, 1419–1425. Wetterneck, T.B., Lapin, J.A., Karsh, B., Beasley, J.W., 2011. Human factors and ergonomics in primary care. In: Carayon, P. (Ed.), Handbook of Human Factors and Ergonomics in Patient Safety, second ed. Taylor and Francis, Boca Raton, FL, pp. 763–775. White, B., 2012. The state of family medicine. Fam. Pract. Manag. 1, 20–26. Wisconsin Research, Education Network. http://www.fammed.wisc.edu/research/wren/ about. Accessed November 30, 2018. Young, G., Zavelina, L., Hooper, V., 2008. Assessment of workload using NASA Task Load Index in perianesthesia nursing. J Perianesth Nurs 23, 102–110. Young, R.A., Burge, S., Kumar, K.A., Wilson, J., 2017. The full scope of family physicians’ work is not reflected by current procedural terminology codes. J. Am. Board Fam. Med. 30, 724–732.
France, D.J., Levin, S., Hemphill, R., Chen, K., Rickard, D., Makowski, R., Jones, I., Aronsky, D., 2005. Emergency physicians’ behaviors and workload in the presence of an electronic whiteboard. Int. J. Med. Inform. 74, 827–837. Gaba, D.M., Lee, T., 1990. Measuring the workload of the anesthesiologist. Anesth. Analg. 71, 354–361. Gawande, A.. Why doctors hate their computers. The New Yorker. https://www.newyork er.com/magazine/2018/11/12/why-doctors-hate-their-computers. December 12, 2018. Guilbert, T.W., Arndt, B., Temte, J., Adams, A., Buckingham, W., Tandias, A., Tomasallo, C., Anderson, H.A., Hanrahan, L.P., 2012. The theory and application of UW eHealth-PHINEX, a clinical electronic health record–public health information exchange. Wis. Med. J. 111 (3), 124–133. Hart, S., Staveland, L., 1988. Development of NASA-TLX (task Load Index): results of empirical and theoretical research. In: Hancock, P., Meshkati, N. (Eds.), Human Mental Workload. North Holland, Amsterdam, pp. 139–183. Helmreich, R.L., 2000. On error management: lessons from aviation. BMJ 320, 781–785. Holden, R.J., Patel, N.R., Scanlon, M.C., Shalaby, T.M., Arnold, J.M., Karsh, B., 2010. Effects of mental demands during dispensing on perceived medication safety and employee well being: a study of workload in pediatric hospital pharmacies. Res. Soc. Adm. Pharm. 6 (4), 293–306. Holden, R.J., Scanlon, M.C., Patel, N.R., Kaushal, R., Escoto, K.H., Brown, R.L., Alper, S. J., Arnold, J.M., Shalaby, T.M., Murkowski, K., Karsh, B.T., 2011. A human factors framework and study of the effect of nursing workload on patient safety and employee quality of working life. BMJ Qual. Saf. 20 (1), 15–24. Holman, G.T., Beasley, J.W., Karsh, B.-T., Stone, J.A., Smith, P.D., Wetterneck, T.B., 2016. The myth of standardized workflow in primary care. J. Am. Med. Inform. Assoc. 23, 29–37. Horner, R.D., Szaflarski, J.P., Ying, J., Meganathan, K., Matthews, G., Schroer, B., Weber, D., Raphaelson, M., 2011. Physician work intensity among medical specialties: emerging evidence on its magnitude and composition. Med. Care 49, 1007–1011. Karsh, B., Holden, R.J., Alper, S.J., Or, K.L., 2006. A human factors engineering paradigm for patient safety: designing to support the performance of the health care professional. Qual. Saf. Health Care 15 (Suppl. I), i59–i65. Katerndahl, D.A., Wood, R., Ja�en, C.R., 2010. A method for estimating relative complexity of ambulatory care. Ann. Fam. Med. 8, 341–347. Leedal, J.M., Smith, A.F., 2005. Methodological approaches to anaesthetists’ workload in the operating theatre. Br. J. Anaesth. 94, 702–709. Longo, L., 2016. Mental workload in medicine: foundations, applications, open problems, challenges and future perspectives. Computer-based medical systems (CBMS) 29th. IEEE International Symposium 30, 24. June. Lowndes, B.R., Forsyth, K.L., Blocker, R.C., Dean, P.G., Truty, M.J., Heller, S.F., Blackmon, S., Hallbeck, M.S., Nelson, H., 2018 Sep 20. NASA-TLX assessment of surgeon workload variation across specialties. Ann. Surg. https://doi.org/10.1097/ SLA.0000000000003058 ([Epub ahead of print]).
8