Measurement Under the Microscope: High Variability and Limited Construct Validity in Emergency Department Patient-Experience Scores

THE PRACTICE OF EMERGENCY MEDICINE/ORIGINAL RESEARCH

Measurement Under the Microscope: High Variability and Limited Construct Validity in Emergency Department Patient-Experience Scores Jesse M. Pines, MD, MBA; Pooja Penninti; Sukayna Alfaraj, MBBS; Jestin N. Carlson, MD, MS; Orion Colfer, MD; Christopher K. Corbit, MD; Arvind Venkat, MD* *Corresponding Author. E-mail: [email protected].

Study objective: We evaluate variability and construct validity in commercially generated patient-experience survey data in a large sample of US emergency departments (EDs). Methods: We used Press Ganey patient-experience data from a national emergency medicine group from 2012 to 2015 across 42 facilities and 242 physicians. We estimated variability as month-to-month changes in percentile scores and through intraclass correlations. Construct validity was assessed with linear regression analysis for monthly facility- and physician-level percentile scores. Results: A total of 1,758 facility-months and 10,328 physician-months of data were included. Across facility-months, 40.8% had greater than 10 points of percentile change, 14.7% changed greater than 20 points, and 4.4% changed greater than 30. Across physician-months, 31.9% changed greater than 20 points, 21.5% changed greater than 30, and 13.6% changed greater than 40. Intraclass correlation estimates demonstrated similar variability; however, this was reduced as data were aggregated over fixed time increments. For facility-level construct validity, several facility factors predicted higher scores: teaching status; more older, male, and discharged patients without Medicaid insurance; lower patient volume; less requirement for physician night coverage; and shorter lengths of stay for discharged patients. For physician-level construct validity, younger physician age, participating in satisfaction training, increasing relative value units per visit, more commercially insured patients, higher computed tomography or magnetic resonance imaging use, working during less crowded times, and fewer night shifts predicted higher scores. Conclusion: In this sample, both physician- and facility-level patient-experience data varied greatly month to month, with physician variability being considerably higher. Facility-level scores have greater construct validity than physicianlevel ones. Optimizing data gathering may reduce variability in ED patient-experience data and better inform decisionmaking, quality measurement, and pay for performance. [Ann Emerg Med. 2017;-:1-10.] Please see page XX for the Editor’s Capsule Summary of this article. 0196-0644/$-see front matter Copyright © 2017 by the American College of Emergency Physicians. https://doi.org/10.1016/j.annemergmed.2017.11.011

INTRODUCTION Background The central role of emergency departments (EDs) is to provide appropriate medical care; however, a secondary goal is to deliver a positive patient experience.1 The concept of measuring and rewarding hospitals and physicians who deliver better patient experience is embraced by providers and by US government entities, such as the Centers for Medicare & Medicaid Services, which has integrated patientexperience data into public reporting and value-based purchasing models for inpatient hospital care since 2007 and 2012, respectively.2,3 In the coming years, patient-experience data will become increasingly important with the implementation of the Medicare Access and CHIP Volume

-,

no.

-

:

-

2017

Reauthorization of 2015.4 ED patient-experience data, although not directly measured in current Centers for Medicare & Medicaid Services value-based purchasing models, has been linked to patients’ assessment of their satisfaction with their overall hospitalization.5 Because quality measurement is increasingly becoming integrated into public reporting and payment policies, ED patient-experience scores will likely become increasingly visible and influential. Currently, ED patient-experience data are used locally to measure and assess physicians and facilities and are sometimes used for physician financial bonuses and remediation, as well as benchmarking and feedback. Yet optimizing ED patient experience is challenging. Emergency physicians do not have established patient-physician Annals of Emergency Medicine 1

Pines et al

Emergency Department Patient Experience Scores

Editor’s Capsule Summary

What is already known on this topic Emergency department (ED) patient satisfaction scores are widely used to assess emergency physicians and EDs despite concern that the scores have limited meaning. What question this study addressed How do physician and facility patient satisfaction scores vary across time? What physician- and facilityspecific factors can predict patient satisfaction scores? What this study adds to our knowledge Patient satisfaction scores varied widely month to month and even year to year. Several factors were found to be associated with scores, although few factors were under the control of either physicians or facilities. How this is relevant to clinical practice Flaws in patient satisfaction assessment limit its usefulness as a feedback mechanism for both physicians and departments.

relationships. They also must diagnose disease and treat and communicate with patients, sometimes with brief interactions, under dire circumstances. Furthermore, some factors that influence ED patient experience are outside of the control of physicians. Optimizing patient experience is difficult when working within systems with long waiting times, high boarding rates, or other hospital inefficiencies.6 Nevertheless, different stakeholders tout ways for physicians to change communication practices, thus improving how patients perceive and ultimately rate their ED experience.7 Importance Despite the charge to improve ED patient experience, issues have been identified with the data. A recent systematic review demonstrated significant variation in the quality of development, as well as psychometric basis, of patientreported experience measures in emergency care.8 There are also similar concerns with the validity of customer-experience data in the business literature outside of medicine.9,10 However, even when surveys are developed with sufficient reliability and validity, another fundamental concern is that data collection processes result in very low numbers of responses and low response rates. The issues are bias and representativeness: data gathered from the few individuals who 2 Annals of Emergency Medicine

choose to answer surveys may not be sufficient to generalize about the care of facilities or individual physicians because they do not reflect the “average” patient experience. The physician community has been critical of patient-experience data: a 2011 information article from the American College of Emergency Physicians raised concerns about data capture, reporting, and use.11 The report described claims from Press Ganey that to judge physician performance, 30 to 50 surveys must be returned. Nevertheless, data are often tabulated with far fewer surveys than that. Furthermore, admitted and transferred patients are not commonly surveyed, which skews responses toward lower-acuity visits in which emergency physicians spend less time and fewer resources. These issues together make it problematic to use experience data to drive managerial decisions and, in particular, compensation or employment decisions. To our knowledge, there have been no external evaluations of ED patient-experience data. Goals of This Investigation In this study, we assessed the variation in and predictors of commercially generated patient-experience survey data in a large sample of emergency physicians and hospitals. Specifically, we assessed month-to-month variability at different levels of data aggregation (ie, monthly to yearly) at the physician and facility level. We also explored how mutable and immutable factors predicted scores to test construct validity. MATERIALS AND METHODS Study Design and Setting This was a retrospective cross-sectional study using data from a national emergency medicine group from January 1, 2012, to December 31, 2015. Operational, physician, facility, and patient-experience data were used for the study. Operational, patient, and physician data were generated by trained billing and coding specialists with relevant certifications, all of whom had ongoing internal audits and oversight. Physician scheduling data are maintained with scheduling software (version 4.9.19.6528; Tangier, Sparks, MD). Patient-experience data were collected by a commercial vendor (Press Ganey, South Bend, IN) and sent to the group for assessment. Data from these multiple sources were combined by information technology staff at the visit level into a single database for research purposes and transferred to Carnegie Mellon University under a data use agreement. This study was approved by the institutional review board at Carnegie Mellon University. Selection of Participants During the study period, the physician group held 104 facility contracts. We limited the sample to facilities with Volume

-,

no.

-

:

-

2017

Pines et al

Emergency Department Patient Experience Scores

greater than or equal to 24 months of continuous and complete data, greater than or equal to 1,000 patient visits per month, and a greater than or equal to 5% admission rate. We also excluded pediatric EDs, given their unique characteristics. Emergency physicians were included with greater than or equal to 36 months of continuous, complete data, greater than or equal to 100 visits per month, a greater than or equal to 5% admission rate, and greater than or equal to 100 monthly clinical hours. We excluded physicians whose patients had an average age of younger than 18 years. These inclusions and exclusions were intended to compare physicians and facilities who practice in similar settings with sufficient data to evaluate across time, with a longer period used for physicians, given the fewer patient surveys likely to be received for each provider when variability and validity were considered. Table E1 (available online at http://www.annemergmed.com) shows the number of facilities and physicians excluded, leading to the final data set of 42 facilities and 242 physicians. Figures E1 and E2 (available online at http://www.annemergmed.com) demonstrate a flow chart for the exclusion of both facilities and physicians to generate the final sample. Methods of Measurement All patients discharged during the study period received either a single mailed paper or electronic survey. Surveys were not administered to admitted or transferred patients, those who left without being seen, those who died in the ED, or those who had rarer exclusions (ie, newborns, prisoners, or those who were sexually assaulted). Patients without a home address or without an e-mail address were not surveyed. The survey instrument included 76 separate multiple-choice questions about aspects of ED care, with 3 to 11 answers each (ie, yes/no/not applicable to Likert scales) and with options for write-in responses. There were standard questions that are included on all surveys. Facilities were able to add a small number of custom questions for internal use only. Surveys administered in relation to a pediatric visit were sent to the attention of the parent or guardian. All patients meeting these criteria were surveyed unless they had been surveyed in the previous 90 days. Survey questions assessed the reason for ED visit, waiting times, pain assessment and management, courtesy and communication by providers, and understanding, as well as overall experience. Four questions focused on individual physicians and were related to courtesy, listening, keeping the patient informed, and concern for comfort. Survey answers created a Press Ganey raw score and were then converted to a percentile rank. Data from write-in responses were not examined in our study. The percentile rank was the primary outcome at 2 levels: physician and facility. Facility monthly percentile rank was Volume

-,

no.

-

:

-

2017

reported by Press Ganey to the group as a 3-month rolling average, using all questions. Physician monthly percentile rank was calculated by the group with raw data from physicianspecific questions from the previous 30 dates of service. Physicians working at more than one facility had a Press Ganey monthly percentile rank reported separately for each facility. For visits involving an advanced practice provider, Press Ganey does not separate physicians versus advanced practice providers. Survey results were attributed to the clinician of record. Therefore, for patients treated by both an advanced practice provider and a physician, the score was attributed to both. In this study, we did not separate consideration of visits involving exclusively physicians versus those involving an advance practice provider; all visits with an included physician as a clinician of record were analyzed together. We studied operational, provider, facility, visit, and patient variables with conceptual relationships to experience scores and used regression analysis to assess construct validity. At the facility level, this included annual visit volume, proportion of physician-hours that were 12 AM to 6 AM, median discharged length of stay, proportion of patients discharged, relative value units per visit (ie, practice intensity), and proportion of patients with computed tomography (CT) or magnetic resonance imaging (MRI). We also assessed ED crowding for each facility at 15-minute increments as median number of patients in the ED per physician working at that increment. This was calculated for this study from the visit and scheduling data in the data set. We also included average physician Press Ganey percentile within each facility, an urban or rural flag based on zip code (US Department of Agriculture classification),12 trauma designation, and teaching facility status, defined as having at least one sponsored graduate medical education program. A subset of included facilities had emergency medicine residency programs or were emergency medicine training sites. Facilitylevel patient characteristics included percentage of male patients, average patient age, and payer source. At the physician level, we included similar operational factors, including proportion of clinical hours 12 AM to 6 AM, discharged length of stay, relative value units per visit, and proportion of patients with CT or MRI. We also applied a similar measure of ED crowding to times when the physician was working. We used years in postresidency practice, physician age, and history of having gone through onetime patient-experience and efficiency training courses. We also used patient population attributes, including proportion of male patients, average patient age, and payer source. Primary Data Analysis The data were tabulated with standard descriptive statistics at the facility and physician level. Physicians who Annals of Emergency Medicine 3

Pines et al

Emergency Department Patient Experience Scores

worked at more than one facility in a month had their monthly Press Ganey percentile ranks blended for each month according to weighting of the number of patients treated at each facility before analysis. To examine variability, we estimated the proportion of scores in which there was a variation of greater than 10%, 20%, and 30% from month to month and an intraclass coefficient variation in Press Ganey percentiles at facility and physician levels, using 1-, 3-, and 6-month increments, as well as 1-year increments. To study individual facility and physician monthly Press Ganey score changes, the absolute monthly changes in percentile scores were tabulated to determine the amount of change (whether positive or negative) in monthly scores. The intraclass correlation coefficient was estimated with the lme4 procedure in R (version 3.32; The R Foundation, Vienna, Austria) with a linear mixed-effects model. The intraclass correlation coefficient tested the between-group variance (facilities or physicians) and within-group variance (facility or physician monthly Press Ganey percentile rank), respectively. The facility (or physician) was the class variable, and the Press Ganey score was represented as a sum of the class variable and a rater-specific random effect in which different raters in different months provided monthly Press Ganey scores for each facility and physician. Given that not all facilities (or physicians) had data for all potential months of the study period (because some facilities and physicians appeared longer in the data set than others after minimum time inclusion criteria), we could not use Cronbach a, or more traditional repeated measures (ANOVA), understanding that mixed-effect models such as intraclass correlation coefficient are preferred over the repeated measure when missing values are dealt with. Construct validity was assessed according to associations and explanatory power of independent variables described above. Facility dummy variables were not used because facility differences have other factors that can be explicitly modeled (eg, average patient age, percentage of male patients, nature of the facility [ie, trauma or teaching]). Given that only 42 facilities met inclusion criteria, adding facility dummies may increase explained variation based on unobserved factors such as cleanliness or nursing interactions, but could reduce the explanatory power of included independent variables that vary by facility. Facility Press Ganey percentile scores and average physician Press Ganey percentile scores were shifted back by 1 month because scores reported in a given month reflect ratings by patients treated predominantly in the previous month. This was empirically verified by correlation before final model construction (data not shown). 4 Annals of Emergency Medicine

We used a 2-stage model to evaluate factors associated with monthly facility Press Ganey percentile rank because there is inherent endogeneity in the facility Press Ganey score that is dependent on the average Press Ganey score of physicians working at a given facility. The 2-stage model is expressed as: Equation 1: facility average physician Press Ganey it ¼ a þ bj  Xjit1 þ bk  Zkit1 þ nit Equation 2: facility Press Ganey it ¼ a þ b1  estimated facility average physician Press Ganeyit þ bj  Xjit1þεit

where i¼facility i, and t¼time period t, Xj are facility explanatory variables, and Zk are physician instrumental variables for the average facility physician Press Ganey score model. Results from the 2-stage least squares multiple regression were estimated with the ivreg, using the R package. Standard regression diagnostics were used. We also tested construct validity of physician Press Ganey percentile ranks. To account for individual physician differences, physician dummy variables were also included. Unlike in the facility model, there were fewer independent variables for modeling that were specific to physicians, and a larger sample of physicians allowed such dummy variable inclusion. Press Ganey scores were lagged by 1 month according to expected predominance of the timeline of responses and empirical correlations at the facility level. The physician Press Ganey model form is as shown below (where i¼physician i, for month t) and X i are the independent variables: Equation 3: physician Press Ganey it ¼ a þ bj  Xjit1 þ εit

All statistical analyses were conducted with R. More detail is available in the Appendix E1, available online at http://www.annemergmed.com. RESULTS Characteristics of Study Subjects During the study period, there were 7.07 million visits, with a mean of 4,006 visits (SD 1,637) and 1,352 physician-hours (SD 577) per month per facility. Length of stay for admitted patients was 5.49 hours (interquartile range 2.02 hours); for discharged patients, 3.10 hours (interquartile range 0.88 hours). For physicians, mean age was 42.5 years (SD 9.6), with an average of 10 years in practice (SD 9), and 93% were board certified. Included Volume

-,

no.

-

:

-

2017

Pines et al

Emergency Department Patient Experience Scores

Volume

-,

no.

-

:

-

2017

A

Absolute Monthly Changes in Facility Press Ganey Percentile Rank

70% % of Observations

60%

59.2%

50%

40% 26.1%

30% 20%

10.4%

10%

3.1%

0.9%

0.3%

30

40

50+

0%

0

10

20

Absolute Month-to-Month Change in Facility Press Ganey Percentile Rank

B

Absolute Monthly Changes in Physician Press Ganey Percentile Rank

50.0%

44.5%

40.0%

% of Observations

physicians had 3.81 million visits, with means of 355 visits (SD 167) and 130.5 clinical hours (SD 24.5) per month. A total of 48% of physicians on average completed onetime patient-experience training at some point during the study period. Table E2 (available online at http://www.annemergmed.com) shows descriptive characteristics of facilities, and Table E3 (available online at http://www.annemergmed.com) shows physician characteristics. Across facility-months, the average absolute monthly change in facility Press Ganey score was 9.1 points. A total of 40.8% facility-months were followed by an absolute change in Press Ganey score of greater than 10 points. In 14.7%, there was a change of 20 points or greater, and 4.4% of facility-months had a change of 30% or more (Figure 1A). Across physician-months, there was an average absolute monthly change in Press Ganey score of 15.8 points. A total of 31.9% of physician-months had an absolute monthly change in Press Ganey score greater than 20 points. In 21.6% of physician-months, there was a change of 30 points or more, and 13.6% of physician-months were followed by a change of 40 points or more (Figure 1B). The facility-month intraclass correlation coefficient was 0.56 (n¼1,758 observations), which increased to 0.64 (n¼135) at the facility-year level. The physician-month intraclass correlation coefficient was 0.34 (n¼10,328 observations), which increased to 0.49 at the physician-year level (n¼968). For both analyses, increasing levels of data aggregation increased intraclass correlation coefficients (Table). Several facility factors were associated with increased monthly facility scores, including higher physician scores, higher proportions of discharged patients, higher proportions of male and older patients, and teaching facility status. Lower facility-level scores were associated with higher patient volumes, greater night hours, longer stays for discharged patients, and more Medicaid patients. Several notable factors were not significant; specifically, the patientsto-physicians ratio (a surrogate for ED crowding) and CT and MRI proportions. The model adjusted R2 was 62%. Figure 2 shows how to interpret proportion changes in model variables. Table E4 (available online at http://www. annemergmed.com) shows the detailed facility model. Several physician factors were associated with higher monthly physician scores, including experience training, relative value units per patient, more commercially insured patients, and higher proportion of patients receiving CT or MRI. Several factors predicted lower scores, including increasing physician age, more night hours, longer stays for discharged patients, and practicing during more crowded periods. The model adjusted R2 was 37%. Figure 3 shows how to interpret proportion changes in significant model

30.0%

23.6%

20.0%

10.3%

10.0%

8.0%

4.2%

3.1%

2.4%

1.8%

1.1%

0.9%

40

50

60

70

80

90

0.0% 0

10

20

30

Absolute Month-to-Month Change in Physician Press Ganey Percentile Rank

Figure 1. Changes in facility and physician Press Ganey percentile rank during the study period. A, Variability in facility Press Ganey percentile rank. B, Variability in physician Press Ganey percentile rank.

variables. Table E5 (available online at http://www. annemergmed.com) shows the detailed physician model. LIMITATIONS There are several limitations in this study. First, we did not calculate precise response rates during the entire study Table. Facility and emergency physician monthly Press Ganey percentile rank intraclass correlation results. Facility Press Ganey Percentile Rank (N[42) Monthly Quarterly Semiannual Annual Emergency physician Press Ganey percentile rank (N[242) Monthly Quarterly Semiannual Annual

No. of Observations for Period

ICC

1,758 579 283 135

0.558 0.598 0.629 0.638

10,328 3,683 1,878 968

0.342 0.388 0.446 0.492

ICC, Intraclass correlation. The table shows the ICC estimate for evaluation of monthly facility and emergency physician Press Ganey percentile rank as calculated on a 1-month, 3-month, 6-month, and yearly level. The data show that variability lessens as data are aggregated over time for both facilities and emergency physicians, but still below an optimal level of 0.7, according to National Quality Forum standards.

Annals of Emergency Medicine 5

Pines et al

Emergency Department Patient Experience Scores

Figure 2. Effect of significant facility independent variables on monthly facility Press Ganey percentile rank. The figure shows the predicted effect of each independent variable on monthly facility Press Ganey percentile rank, along with 95% confidence intervals, holding all other independent variables in the multivariable regression model constant. *A one hour increase in mean patient length of stay in the ED of patients who are discharged from the facility, holding all other factors constant, would be associated with the average monthly Press Ganey percentile rank for the facility decreasing between 3.1 to 8.9 points (95% confidence interval). **A one point increase in the percentage of patients discharged in a month at an ED, holding all other factors constant, would be associated with the average monthly facility Press Ganey percentile rank increasing between 0.5 to 1.0 points (95% confidence interval).

and link them to corresponding facilities, physicians, and months. This limited our conclusion that low response rates, which contribute to higher sampling error, were the prime contributor to high variability and low intraclass correlation coefficients. However, a 6-month sample demonstrated average facility response rates from 3.6% to 16.0%. Second, our sample included 42 EDs during 4 years (<1% of US EDs), and data were used from a single

vendor. Results may have been different in other hospitals or with other vendors. Third, we did not measure important factors closely related to patient experience, such as listening skills or whether physicians or other staff communicated effectively and empathetically.13 Similarly, facility-level attributes that may affect experience, such as facility design or culture, were not measured.

Figure 3. Effect of significant physician independent variables on monthly physician Press Ganey percentile rank. The figure shows the predicted effect of each independent variable on monthly physician Press Ganey percentile rank, along with 95% confidence intervals, holding all other independent variables in the multivariable regression model constant. *A one hour increase in the mean patient length of stay of patients assigned to a physician who are discharged from the ED, holding all other factors constant, would be associated with the average monthly Press Ganey percentile rank for the physician decreasing between 4.4 to 7.4 points (95% confidence interval). **A one point increase in the mean RVUs/hour generated by the physician, holding all other factors constant, would be associated with the average monthly physician Press Ganey percentile rank increasing between 3.4 and 10.3 points (95% confidence interval).

6 Annals of Emergency Medicine

Volume

-,

no.

-

:

-

2017

Pines et al

Emergency Department Patient Experience Scores

Fourth, there are limitations with using intraclass correlation coefficients, which assumed we were measuring precisely the same sample during the study period to generate a true “reliability” estimate. This implies that patient profiles and facility factors are the same across months, which was not true in our data. However, this is more likely for facilities with a larger and possibly more similar patient pool over time as compared with those of individual physicians. This would be especially true for annual scores, given that the demographics of patients and external facility factors during an entire year vary less, as observed in higher annual intraclass correlation coefficient estimates. Fifth, when it came to assessing construct validity, we included variables according to their availability and whether they were conceptually related to patient-experience data, based on the clinical experience of our team and previous studies.5,6 Other data—unavailable or untested—may have also confounded the construct validity analysis. Sixth, because data gathered were from only discharge patients with e-mail addresses or mailing addresses, it is possible that patient experience from other groups of patients, such as those admitted, would yield different results. Seventh, there were also attribution issues, particularly when an assessment of an advanced practice provider or resident may have been assigned to the attending physician. This would bias toward greater variation across months. However, our data were what is reported back to physicians. Therefore, biases toward greater variation is implicit in how the data are processed. DISCUSSION During the past decade, there has been increased focus on quality measures in emergency care.14 One such focus has been on optimizing patient experience, which is a major goal of providers and facilities and a basis for competition. However, ensuring that data are representative and accurate is vital. In this study, we used a large sample of physicians and facilities to assess these concepts. To quantify variability, we assessed the month-to-month variation in Press Ganey percentile ranks through simple descriptive statistics of how much a facility’s or physician’s score changed from one month to the next. We found that facility and physician percentile ranks changed considerably across months: a 9-point average change for facilities and a 16-point change for physicians. It was not uncommon for scores to demonstrate a large shift in which one month a facility could be performing well and the next it could be rated as a poor facility. This was accentuated among physician scores. Conceptually, it would be expected that scores should change, particularly with changes in care Volume

-,

no.

-

:

-

2017

delivery. Yet in reality, facilities and physicians tend to deliver similar care, particularly when any particular month is compared with the one immediately before it. Therefore, if intrinsic attributes of the care experience are adequately captured by the survey process, then scores should not demonstrate such great variability month to month. We also used an intraclass correlation coefficient, a statistic that describes how much scores vary, and tested it in month-to-month to year-to-year comparisons. This was intended to test whether greater data aggregation smooths out scores over time. According to the National Quality Forum, a reliable quality measure should meet a minimum intraclass correlation coefficient of 0.70.15 However, we used intraclass correlation coefficient in a slightly different way. Conceptually, reliability for a quality measure means measuring the same data twice. In our study, intraclass correlation coefficient should be viewed as another measure of variability rather than reliability. However, with this standard, even facility-level data demonstrated a marginal intraclass correlation coefficient (less than 0.70): 0.56 for monthly data, which increased when aggregated, demonstrating that combining data during greater periods does have a beneficial smoothing effect. Physician-level data had a low intraclass correlation coefficient of 0.34, and even with data aggregation during a year, the variability was still very high. One potential explanation for greater variability among physicians may be an effect of the information volume: there are fewer questions about physicians—only 4 per survey—but many more about the facility. All this variation represents concerning measurement issues in the survey itself, the data collection process, or how the survey percentile ranks are generated. To further demonstrate why this may be occurring, we plotted the average facility and physician monthly Press Ganey percentile ranks against their SD (Figure 4A and B, respectively). We observed that facilities and physicians at the extremes must change their scores considerably to move their percentile rank, whereas the physicians near the 50th percentile can change minimally to have great swings in percentile ranks, which are unlikely to be clinically meaningful. This means the data are, in effect, bunched. Forcing a percentile rank on raw scores that are minimally different may also explain some of the variability, at least in part. One potential solution for this would be to present raw scores instead of percentile rank. For construct validity, several factors conceptually associated with experience predicted facility- and physicianlevel scores. Yet facility models were considerably more predictive than physician models. Some predictive factors were mutable—organizations may be able to change through management interventions—whereas others were Annals of Emergency Medicine 7

Pines et al

Emergency Department Patient Experience Scores

Figure 4. A, Plot of mean monthly facility Press Ganey percentile rank versus SD of mean monthly facility Press Ganey percentile rank. The 42 facilities (each represented by a single data point) in the chart show that those with low mean score (near 0%) or high mean score (near 100%) have lower Press Ganey variability. Facilities with percentile rank near 50% have higher volatility in their ranking, in which small changes in mean score could make a large difference in the percentile rank. B, Plot of mean monthly physician Press Ganey percentile rank versus SD of mean monthly physician Press Ganey percentile rank. The nature and structure of the chart for 242 physicians (each represented by a single data point) are similar to those of the facility chart, showing physicians with low mean score (near 0%) or high mean score (near 100%) having lower Press Ganey variability. Physicians with percentile rank near 50% have much higher volatility in their ranking, in which small changes in mean ratings could make a large difference in the percentile rank.

immutable, more of an inherent attribute of the facility or patient population. For facility scores, one immutable factor that increased scores was teaching facility status, which may be explained by differences in how patients perceive residents or fellows, that academic facilities actually produce better experiences, or a halo effect. Patient factors also predicted better scores, in particular facilities with older, male, non8 Annals of Emergency Medicine

Medicaid patients scoring better. Similar patient-level data have previously demonstrated trends toward higher scores in older patients with private insurance.16 The sole measured and mutable factor that predicted higher facility score was shorter discharged length of stay, a phenomenon observed in another study linking ED crowding to experience.17 However, ED crowding across facilities did not decrease

Volume

-,

no.

-

:

-

2017

Pines et al

Emergency Department Patient Experience Scores

scores, suggesting that more crowded facilities may not inherently produce worse experiences after controlling for length of stay, which is how patients experience crowding. Overall, we can conclude there is reasonable construct validity in facility scores. However, the degree to which scores can be changed through interventions or whether the scores are more a reflection of facility characteristics is an open question. Consideration could be given to risk-adjust for immutable patient and fixed facility factors that predict lower scores, such as Medicaid case mix. Physician-level models were less predictive, suggesting that construct validity for physician scores was less, and arguably marginal with the low R2. Nevertheless, several factors predicted higher scores, such as physicians’ having undergone patient experience training, and a higher intensity of care, as well as higher use of CT or MRI. This suggests that despite high variability, physician scores may be, to some degree, a mutable concept.18 To some degree, this finding confirms those of a previous study in abdominal pain, in which patients who received CTs reported greater confidence in their care regardless of test results.19 Increasing physician age also predicted poorer scores, suggesting that communication practices may change with age or that there may be generational or training differences. However, more research is needed to confirm this finding. Several factors related to physician practice also predicted lower scores, including working more night shifts and working during more crowded shifts. Together this suggests that a similar approach to risk adjustment could be useful to better compare physicians, assuming a reliable data collection method were implemented. Our findings have important implications for the use of experience data. Broadly, we agree that the concept of measuring patient experience is important for many reasons but recommend different approaches to reduce the large observed variability. Reporting and acting on unsound data undermine the intent of the exercise and may lead to uninformed, ineffective, or, at worst, unfair management practices or payment determinations. Our work also raises specific concerns about data such as the Hospital Consumer Assessment of Healthcare Providers and Systems to drive Medicare payments, which is measured through a similar data-gathering methodology. In the 2014 National Quality Forum review of the Hospital Consumer Assessment of Healthcare Providers and Systems, interitem correlations were used to assess reliability of the measure, rather than the overall measure of score variability we used in our study.20 Future work in assessing patient-experience data should consider focusing on the effect of response rates and testing the effect on variability and reliability results at the score level. Volume

-,

no.

-

:

-

2017

Despite current limitations, several approaches may improve response rates. One approach is a shorter survey, administered immediately after ED departure by e-mail or text.21 However, previous work has found that this sort of methodology generates a response rate of only approximately 26%.22 This suggests other efforts may be necessary to increase responses, such as providing financial incentives to patients or providing discounts on bills for filling out surveys. However, providing such incentives could potentially affect the validity of the survey information. Alternatively, further investments could be made in ensuring that each patient fills out a survey through follow-up telephone calls, which may also help identify patient safety issues, as well as improve perceptions of ED care.23 EDs may also consider closer inspection of write-in comments or qualitative feedback from Web sites like Yelp.24 Finally, the use of percentile ranks for raw scores that are so closely spaced should be reconsidered. In conclusion, we found great month-to-month and even year-to-year variation in commercially generated patient-experience data in this sample. This is concerning, given the widespread use of such data across emergency medicine. Construct validity was greater for facility-level than physician-level data, and many factors predicted scores, some of which are not under management control. A closer inspection of the process of gathering patient experience data by survey vendors is warranted to help improve the process of measuring ED patient experience. The authors acknowledge Susan Fix, JD, of US Acute Care Solutions for her work on the regulatory aspects of this research collaboration; Paul Dietzen and Jesse Eterovich, BS, of US Acute Care Solutions for their work in compiling the data of this study; Allison Rittmaier, AS, of US Acute Care Solutions for her assistance in understanding the patient experience survey process; and Dominic Bagnoli, MD, Michael Osmundson, MD, James Augustine, MD, Amer Aldeen, MD, and the leadership of US Acute Care Solutions for their support of this research project. Supervising editor: Stephen Schenkel, MD, MPP Author affiliations: From the Departments of Emergency Medicine and Health Policy and Management, The George Washington University School of Medicine and Health Sciences, Washington, DC (Pines); Carnegie Mellon University, Pittsburgh, PA (Penninti); the Department of Emergency Medicine, College of Medicine, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia (Alfaraj); the Department of Emergency Medicine, Allegheny Health Network, Pittsburgh, PA (Carlson, Venkat); and US Acute Care Solutions, Canton, OH (Carlson, Colfer, Corbit, Venkat). Author contributions: JMP, JNC, OC, CKC, and AV conceived the study. JNC, CKC, and AV supervised data collection. PP performed the statistical analysis. JMP, PP, SA, and AV drafted the article, and Annals of Emergency Medicine 9

Pines et al

Emergency Department Patient Experience Scores all authors contributed substantially to its revision. AV takes responsibility for the paper as a whole. 8.

All authors attest to meeting the four ICMJE.org authorship criteria: (1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND (2) Drafting the work or revising it critically for important intellectual content; AND (3) Final approval of the version to be published; AND (4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org). The authors have stated that no such relationships exist. Publication dates: Received for publication May 25, 2017. Revision received September 27, 2017. Accepted for publication November 9, 2017.

9.

10.

11.

12.

13.

14.

US Acute Care Solutions provided the data for this study to Carnegie Mellon University under a data use agreement. US Acute Care Solutions does not exercise any control over the analysis, conclusions, or decision to publish the analysis presented.

15.

REFERENCES

16.

1. Pines JM, Lotrecchiano GR, Zocchi MS, et al. A conceptual model for episodes of acute, unscheduled care. Ann Emerg Med. 2016;68:484-491.e3. 2. Centers for Medicare & Medicaid Services. Hospital value-based purchasing. Available at: https://www.cms.gov/Outreach-andEducation/Medicare-Learning-Network-MLN/MLNProducts/ downloads/Hospital_VBPurchasing_Fact_Sheet_ICN907664.pdf. Accessed September 24, 2017. 3. Pines JM, McStay F, George M, et al. Aligning payment reform and delivery innovation in emergency care. Am J Manag Care. 2016;22:515-518. 4. Centers for Medicare & Medicaid Services. The merit-based incentive payment system: quality performance category. Available at: https:// www.cms.gov/Medicare/Quality-Initiatives-Patient-AssessmentInstruments/Value-Based-Programs/MACRA-MIPS-and-APMs/QualityPerformance-Category-training-slide-deck.pdf. Accessed September 17, 2017. 5. Pines JM, Iyer S, Disbot M, et al. The effect of emergency department crowding on patient satisfaction for admitted patients. Acad Emerg Med. 2008;15:825-831. 6. Tekwani KL, Kerem Y, Mistry CD, et al. Emergency department crowding is associated with reduced satisfaction scores in patients discharged from the emergency department. West J Emerg Med. 2013;14:11-15. 7. Gamble M. Boosting patient satisfaction in the ED: what hospitals should and shouldn’t do. Becker’s Hospital Review. 2013. Available at: http://www.beckershospitalreview.com/hospitalmanagement-administration/boosting-patient-satisfaction-in-the-ed-

10 Annals of Emergency Medicine

17.

18.

19.

20.

21.

22.

23.

24.

what-hospitals-should-and-shouldn-t-do.html. Accessed September 24, 2017. Male L, Noble A, Atkinson J, et al. Measuring patient experience: a systematic review to evaluate psychometric properties of patient reported experience measures (PREMs) for emergency care service provision. Int J Qual Health Care. 2017;29:314-326. Parasuraman A, Zeithaml VA, Berry LL. Reassessment of expectations as a comparison standard in measuring service quality: implications for further research. J Marketing. 1994;58:111-124. Johnson MD, Fornell C. A framework for comparing customer satisfaction across individuals and product categories. J Econ Psychol. 1991;12:267-286. Members of the ACEP Emergency Medicine Practice Committee. Patient satisfaction. Available at: https://www.acep.org/ patientsatisfaction/. Accessed September 24, 2017. United States Department of Agriculture. The United States Department of Agriculture rural-urban continuum codes. Available at: https://www.ers.usda.gov/data-products/rural-urban-continuumcodes/. Accessed September 24, 2017. Locke R, Stefano M, Koster A, et al. Optimizing patient/caregiver satisfaction through quality of communication in the pediatric emergency department. Pediatr Emerg Care. 2011;27:1016-1021. Schuur JD, Hsia RY, Burstin H, et al. Quality measurement in the emergency department: past and future. Health Aff (Millwood). 2013;32:2129-2138. National Quality Forum. Guidance for measure testing and evaluating scientific acceptability of measure properties. The National Quality Forum, 2011. Available at: http://www.qualityforum.org/Measuring_ Performance/Submitting_Standards/Measure_Evaluation_Criteria. aspx. Accessed September 24, 2017. Handel DA, French LK, Nichol J, et al. Associations between patient and emergency department operational characteristics and patient satisfaction scores in an adult population. Ann Emerg Med. 2014;64:604-608. Morgan MW, Salzman JG, LeFevere RC, et al. Demographic, operational, and healthcare utilization factors associated with emergency department patient satisfaction. West J Emerg Med. 2015;16:516-526. Boissy A, Windover AK, Bokar D, et al. Communication skills training for physicians improves patient satisfaction. J Gen Intern Med. 2016;31:755-761. Baumann BM, Chen EH, Mills AM, et al. Patient perceptions of computed tomographic imaging and their understanding of radiation risk and exposure. Ann Emerg Med. 2011;58:1-7.e2. National Quality Forum. NQF-endorsed measures for person and family centered care. Available at: http://www.qualityforum.org/projects/ person_family_centered_care/. Accessed September 24, 2017. Strickler JC, Lopiano KK. Satisfaction data collected by e-mail and smartphone for emergency department patients: how do responders compare with nonresponders? J Nurs Adm. 2016;46:592-598. Glickman SW, Mehrotra A, Shea CM, et al. A patient reported approach to identify medical errors and improve patient safety in the emergency department. J Patient Saf. 2016; https://doi.org/10.1097/PTS. 0000000000000287. Guss DA, Leland H, Castillo EM. The impact of post-discharge patient call back on patient satisfaction in two academic emergency departments. J Emerg Med. 2013;44:236-241. Kilaru AS, Meisel ZF, Paciotti B, et al. What do patients say about emergency departments in online reviews? a qualitative study. BMJ Qual Saf. 2016;25:14-24.

Volume

-,

no.

-

:

-

2017

Pines et al

Emergency Department Patient Experience Scores

APPENDIX E1 ICC analysis Facility Press Ganey percentile score variability was tested with an ICC estimate using the procedure lme4 with a linear mixed-effects model, using the R statistical package. The methodology tested the between-group variance (facilities) and within-group variance (facility monthly Press Ganey percentile rank). ICC was estimated for monthly, quarterly, semiannual, and annual Press Ganey scores for 42 facilities. Because Press Ganey scores were compared across periods and potentially different patient populations who responded to a Press Ganey survey each period, it can be expected that there will be variation in Press Ganey scores because the measurement does not reflect the same responding population. However, because the sample size for a facility is fairly large (with >3,000 responses for a facility with 40,000 patients per annum, with a 75% discharge rate and 10% survey response), the assumption of similar population and similar operating variables is more likely on an annual basis. REGRESSION ANALYSIS FOR CONSTRUCT VALIDITY We developed a model to link various operational, physician, and patient variables with monthly facility Press Ganey percentile ranks. Specifically, we were interested in determining the key factors that could be driving the variation in monthly facility Press Ganey percentile ranks. Certain relevant facility-specific factors were not available (including nursing, cleanliness, and interaction nature), and facility dummy variables were not used to adjust for this because facility differences have other factors that can be explicitly modeled (eg, average patient age, percentage of male patients visiting the facility, nature of the facility such as trauma or residency/teaching). Given that only 42 facilities met inclusion criteria, adding facility dummies could have increased the explained variation, but could have reduced the explanatory power of the independent variables that vary by facility. Because physician performance, and therefore physician Press Ganey percentile rank, would affect a facility Press Ganey percentile rank, the average Press Ganey percentile rank of physicians working at a facility would be an endogenous variable that would need to be accounted for in the modeling. We therefore performed a 2-stage multiple regression model to evaluate factors associated with monthly facility Press Ganey percentile rank. The 2-stage model can be expressed as: Equation 1: facility average physician Press Ganeyit ¼ a þ bj  Xjit1 þ bk  Zkit1þnit Volume

-,

no.

-

:

-

2017

Equation 2: facility Press Ganeyit ¼ a þ b1  estimated facility average physician Press Ganeyit þ bj  Xjit1þεit

where i¼facility i, and t¼time period t, Xj are facility explanatory variables, and Zk are the instrumental variables for the average facility physician Press Ganey percentile rank model. In the second equation, the average facility physician Press Ganey percentile rank (in the facility) is endogenous and is therefore replaced with the estimated value from equation 1. Both the facility Press Ganey scores and average physician Press Ganey scores are shifted back by 1 month because it was assumed that the score reported in a given month reflected ratings by patients of conditions at that facility predominantly in the previous month. This was empirically verified by estimating correlations of lagged terms to Press Ganey score before final model construction (data not shown). The instruments (ie, the variables that affect the facility Press Ganey score through the average physician Press Ganey score) were average physician age at the facility, percentage at the facility who were male physicians, percentage of physicians at the facility who were white, percentage of physicians at the facility who had completed satisfaction training, and percentage of physicians at the facility who had completed efficiency training. A 2-stage least squares model was estimated for the above equations, in which model form is as shown below, where i¼facility i, for month t STAGE 1: FACILITY AVERAGE PHYSICIAN PRESS GANEY PERCENTILE RANK MODEL The first-stage model included instrumental variables indicated in italics below. Average physician Press Ganey percentile rank within facilityit¼aþb21Average Physician Age it–1þb22% physicians w/satisfaction trainingit–1þb23% physicians w/ productivity trainingit–1þb23% physicians w/efficiency trainingit–1þb23% male physiciansit-1þb23white physiciansit–1þb2ED patient volume it-1þb3ED % night hoursit–1þb4LOS dischargedit–1þb5% dischargedit–1þb6RVU/patientit–1þb7patient/ physician ratio at admissionit–1þb8 %Medicaidit–1þb9% uninsuredit–1þb10% commercial insurance it–1þb11male patientsit–1þb12average patient ageit–1þb13urban indicatoriþb14% CT/MRI proceduresit–1þb15itrauma indicatori–1þb16iteaching indicatori–1þuit LOS, Length of stay; RVU, relative value units. Annals of Emergency Medicine 10.e1

Pines et al

Emergency Department Patient Experience Scores

STAGE 2: FACILITY PRESS GANEY SCORE MODEL The second-stage model was built with monthly facility Press Ganey percentile rank versus monthly operational and patient variables and the estimated average physician Press Ganey percentile rank from the first-stage model (predicted values). The model form is as shown below (where i¼facility i, for month t). Facility Press Ganeyit¼aþb1estimated average physician facility Press Ganeyitþb2ED patient volumeit–1þb3ED % night hoursit–1þb4LOS dischargedit–1þb5% dischargedit–1þb6RVU/ patientit–1þb7patient/physician ratio at admissionit–1þb8%Medicaidit–1þb9% uninsuredit–1þb10% commercial insuranceit–1þb11male patientsit–1þb12average patient ageit–1þb13urban indicatoriþb14% CT/MRI proceduresit–1þb15itrauma indicatori–1þb16iteaching indicatori–1þvit Results from the 2SLS multiple regression were estimated with the ivreg, using the R package. Physician Press Ganey percentile rank variability was tested with an ICC estimated using the procedure lme4, with a linear mixed-effects model using the R statistical package. The methodology tested the between-group variance (physicians) and within-group variance (physician monthly Press Ganey percentile rank). We developed a multiple regression model to link various operational, physician, and patient variables with monthly physician Press Ganey percentile ranks. Specifically, we were interested in determining the key factors that could be driving the variation in monthly physician Press Ganey percentile ranks. Physician Press Ganey percentile rank was modeled to be a function of various physician, patient, and facility operational variables. To account for individual physician differences, physician dummy variables were also included in the model. Unlike in the facility model, there were fewer independent variables for modeling that were specific to physicians and a larger sample of physicians allowing such dummy variable inclusion. Once again, because Press Ganey surveys for a physician in a given month are from discharged patients from one or more months ago, the explanatory variables were lagged by 1 month according to expected

10.e2 Annals of Emergency Medicine

predominance of timeline of responses and the previous empirical correlation at the facility level noted above. The physician Press Ganey model form is as shown below (where i¼physician i, for month t): Physician Press Ganeyit¼aþb1physician ageit–1þb2years in practiceit-1þb3satisfaction training indicatorit–1þb4efficiency training indicatorit–1þb5% night hoursit–1þb6LOS dischargedit–1þb7RVU per patientit–1þb8patient/physician ratio at admissionit–1þb9% Medicaidit–1þb10% uninsuredit–1þb11% commercial insuranceit–1þb12% male patientsit–1þb13average patient ageit–1þb14% CT/MRI proceduresit–1þb2iphysician dummyiþεit Results were then calculated for the multiple regression model. PROCESS OF PRESS GANEY PERCENTILE RANK CALCULATION ON FACILITY AND PHYSICIAN LEVEL The monthly facility score is calculated by Press Ganey according to a rolling 3-month calculation. It is calculated with all the questions on the survey. The facility score is a mean based on all questions on surveys received with dates of service during the previous 3 months. The facility mean score is then ranked in a cohort of facilities with similar volumes to obtain the percentile ranking. There are no accommodations for outlying data. All data are used in calculating scores. The monthly physician scores are calculated with only the 4 standard clinician questions on the survey. Press Ganey provides survey raw data to this emergency medicine group, and the providers scores are calculated within the national group, using the most recent 30 dates of service for each clinician. The mean score is then ranked in a cohort of facilities with similar volumes to obtain the percentile ranking. It is quite variable in regard to the number of surveys returned for a clinician’s patients during the previous month. There is no weighting based on number of patients treated during that 30 dates of service. There is an exclusion for a new calculation for the physician if fewer than 10 surveys are returned by patients from the last 30 days of service.

Volume

-,

no.

-

:

-

2017

Pines et al

Emergency Department Patient Experience Scores

Figure E1. Facility selection flow chart. There was an average of 45 months of data per facility out of 48 months in the data set. The order of applied data filters will not change the final total count because these conditions are applied as an “and” condition. Actual influence of each variable on “exclusion” count is provided in Table E1.

Figure E2. Physician selection flow chart. There was an average of 43 months of data per physician out of 48 months in the data set. The order of applied data filters will not change the final total count because these conditions are applied as an “and” condition. Actual influence of each variable on “exclusion” count is provided in Table E1. Volume

-,

no.

-

:

-

2017

Annals of Emergency Medicine 10.e3

Pines et al

Emergency Department Patient Experience Scores Table E1. Inclusion and exclusion of facilities and emergency physicians according to study criteria. Inclusion-Exclusion Criteria

Met Inclusion Criterion (N)

Met Exclusion Criterion (N)

104 56 59 84 82 100 42

0 48 45 20 22 4 62

1,669 373 1,134 902 1,246 1,242 242

0 1,296 535 767 423 427 1,427

Facilities Initial full data set, 2012–2015 24 mo of full data Average monthly visits of at least 1,000 Average monthly visit admission rate of at least 5% Facilities with CT and MRI use data available Average patient age at facility 18 y Final included facilities Emergency physicians Initial full data set, 2012–2015 36 mo of full data Average monthly visits of at least 100 Average monthly hours worked of at least 100 Average patient age of at least 18 y Average monthly visit admission rate of at least 5% Final included emergency physicians

Table E2. Characteristics of included facilities (N¼42). Facility Independent Variable Monthly patient visits Monthly physician hours Percentage of monthly physician hours between 12 AM and 6 AM, % Length of stay admission visits, h Length of stay discharge visits, h Percentage of monthly visits Admission Discharge Medicaid Uninsured Commercial insurance Medicare Age of patients visiting facility, y Median patients in facility/physician working ratio (15-min increments) Monthly percentage of patients visiting facility, male patients RVUs/h RVUs/visit Patients/h Monthly facility Press Ganey percentile rank Monthly visits with CT or MRI performed, %

Interquartile Range 2,470 845 4.2

Median

Mean

SD

3,707 1,202 17.3

4,006 1,352 17.9

1,637 577 3.9

2.02 0.88

5.49 3.10

5.96 3.12

2.27 0.68

8.20 9.10 13.7 13.3 9.9 13.5 4.48 3.50

17.4 79.3 28.08 15.38 25.44 28.22 41.51 10.50

18.0 78.5 27.78 18.48 25.82 28.11 41.55 10.73

7.5 7.6 10.18 9.90 7.48 10.00 4.16 3.28

3.0

43.7

43.8

2.3

2.91 0.42 0.88 42.5 18.9

11.31 3.83 2.95 55.0 13.5

11.66 3.86 3.04 54.1 11.2

2.44 0.28 0.71 26.3 10.4

Descriptive classifications of included facilities Facilities Facilities Facilities Facilities

Included facilities, %

classified as urban EDs classified as trauma EDs that were emergency medicine residency training sites (primary or primary community site) that were teaching hospitals (at least one sponsored graduate medical education program)

50.4 29.3 15.9 18.5

Not all of the above variables were included in regression models. Data are provided for understanding of included facilities and emergency physicians.

10.e4 Annals of Emergency Medicine

Volume

-,

no.

-

:

-

2017

Pines et al

Emergency Department Patient Experience Scores

Table E3. Characteristics of included emergency physicians (N¼242). Emergency Physician Independent Variable Physician age, y Years in practice (since residency completion) Emergency medicine board certification (percentage of physicians on a monthly basis), % Percentage of physicians on a monthly basis completed patient-experience training, % Percentage of physicians on a monthly basis completed efficiency training, % Number of facilities at which physicians worked on a monthly basis Monthly visits Monthly hours worked Monthly percentage of clinical hours worked 12 AM to 6 AM, % Length of stay of visits, admission on a monthly basis, h Length of stay of visits, discharge on a monthly basis, h Monthly physician Press Ganey percentile rank Monthly percentage of patient visits, % Admission Discharged Medicaid Uninsured Commercial insurance Medicare Male patients With CT or MRI performed Median patients in facility/physician working ratio (15-min increments) at times physician working Age of patients treated by physician in a month, y Monthly RVUs per patient visit

Interquartile Range

Median

Mean

SD

15.00 14.00 –*

40.00 7.00 100

42.48 10.03 93

9.61 9.31 26

–

–

48

–

–

–

41

–

1.00

1.00

1.45

0.82

127.00 28.20 16.3

337.00 130.00 12.5

355.20 130.50 18.4

166.91 24.51 19.2

2.10

5.43

6.03

2.67

1.24

3.13

3.26

0.90

75.00

60.00

56.77

36.18

12.9 14 15.10 14.4 10.2 13.1 5 20.4 3.83

17 80 26.86 15.82 25.66 26.86 43 11.8 10.50

19 78 26.93 18.48 26.14 26.93 44 11.9 10.91

10 11 9.88 9.43 7.12 9.88 4 11.6 3.49

7.25 0.59

41.97 3.90

42.10 3.93

5.23 0.41

Not all of the above variables are included in regression models. Data are provided for understanding of included facilities and emergency physicians. *Dashes indicate that there are no calculated values for these variables.

Volume

-,

no.

-

:

-

2017

Annals of Emergency Medicine 10.e5

Pines et al

Emergency Department Patient Experience Scores

Table E4. Multiple regression model of factors associated with monthly facility Press Ganey percentile (n¼1,758 facility-months). Independent Variable Predicted average physician Press Ganey percentile rank in facility Volume of patient visits to facility Percentage of clinical hours in facility 12 AM to 6 AM of total clinical hours in facility Length of stay discharged Percentage of patient visits to facility discharged RVUs per patient visit to facility Patient arrivals/physician ratio Percentage Medicaid patients Percentage uninsured patients Percentage commercial patients Percentage male patients Patient age, y Percentage patient visits with CT or MRI performed Urban ED Trauma center Teaching facility

Estimate

Standard Error

T Value

95% Lower Confidence Level

95% Upper Confidence Level

1.049

0.077

13.573

0.897

1.200

(0.001)* (53.024)

0.000 13.937

(2.881) (3.805)

(0.002) (80.339)

(0.000) (25.709)

(5.992) 74.025 0.962 0.303 (0.278) (0.107) (0.017) 71.599 0.678 (3.340) (0.331) 1.403 4.766

1.488 13.056 2.947 0.205 0.069 0.070 0.092 20.715 0.172 4.232 0.933 1.207 1.771

(4.026) 5.670 0.327 1.477 (4.034) (1.524) (0.188) 3.456 3.947 (0.789) (0.354) 1.162 2.692

(8.909) 48.436 (4.814) (0.099) (0.413) (0.244) (0.199) 30.998 0.342 (11.634) (2.160) (0.963) 1.296

(3.075) 99.614 6.739 0.704 (0.143) 0.031 0.164 112.201 1.015 4.954 1.498 3.770 8.236

Model characteristics included the following: residual standard error 16.381,694; R2 0.6219; adjusted R2 0.6183; Wald test 129.216, *Numbers in parenthesis indicate a negative value whereas those not in parenthesis indicate a positive value.

1,694;

P<.001.

Table E5. Multiple regression model of factors associated with monthly physician Press Ganey percentile rank (N¼10,328 physicianmonths). Independent Variable

Estimate

Standard Error

T Value

95% Lower Confidence Level

95% Upper Confidence Level

Physician age Years in practice Satisfaction training Efficiency training Percentage of clinical hours between 12 AM and 6 AM of all clinical hours worked Length of stay discharged Percentage of patient visits discharged RVUs/patient Patient arrivals/physician ratio Percentage Medicaid patients Percentage uninsured patients Percentage commercial patients Percentage male patients Patient age, y Percentage patient visits with CT or MRI performed

(3.110)* (0.121) 8.045 1.266 (20.575)

0.734 0.704 1.638 1.540 3.146

(4.234) (0.172) 4.912 0.822 (6.540)

(4.550) (1.501) 4.834 (1.752) (26.742)

(1.670) 1.258 11.255 4.284 (14.408)

(6.034) 7.488 6.870 (0.322) 0.023 0.188 0.248 (8.603) (0.016) 15.468

0.695 8.680 1.743 0.137 0.097 0.101 0.090 9.613 0.167 2.940

(8.687) 0.863 3.942 (2.356) 0.237 1.868 2.771 (0.895) (0.096) 5.261

(7.396) (9.527) 3.453 (0.590) (0.167) (0.009) 0.073 (27.446) (0.343) 9.705

(4.672) 24.502 10.286 (0.054) 0.212 0.385 0.424 10.241 0.311 21.231

Estimates of physician dummy variables are not included in the table, although they are included in the model. Model characteristics included the following: residual standard error 28.7110,114; multiple R2 0.3858; adjusted R2 0.3703; F24.82; DF 256, 10114; P<.0001. *Numbers in parenthesis indicate a negative value whereas those not in parenthesis indicate a positive value.

10.e6 Annals of Emergency Medicine

Volume

-,

no.

-

:

-

2017