- Email: [email protected]
Implementing CQI Projects in Hospitals
THE JOINT COMMISSION Of interest to administrators and quality improvement professionals, the results of this study are among the first to examine costs of using CQI techniques in more than one hospital. QUALITY MANAGEMENT/IMPROVEMENT PROGRAMS
Implementing CQI Projects in Hospitals ANNE SALES, PHD IRA MOSCOVICE, PHD NICOLE LURIE, MD, MSPH
he application of quality improvement (QI) approaches such as continuous quality improvement (CQI) and total quality management (TQM) to health care organizations has been a relatively recent phenomenon.1,2 Quality assurance approaches, still dominant in health care organizations in the United States during the 1980s, were often characterized as highly punitive, with an emphasis on identifying individuals who were thought to cause problems in quality of care rather than attempting to find sources of problems in the systems of care. CQI (or TQM) advocates suggested that these approaches offered a more effective way to approach problems in quality of care and a more humane method of dealing with health care providers. CQI has been a mainstay of quality management in health
T
care organizations since the early 1990s, when the tools and techniques of CQI were strongly encouraged as a component for accreditation by the Joint Commission on Accreditation of Healthcare Organizations (Oakbrook Terrace, Ill).3 Concurrent with this relatively new approach to managing quality of care in hospitals in the United States, in 1989 the Robert Wood Johnson Foundation (RWJF; Princeton, NJ) began a multiyear program aimed at helping hospitals grapple with issues related to quality of care. The intent of the program was to facilitate new models and methods of responding to quality management, and a prime motivator for the program was a model of hospital consortia organized around quality of care issues in the Netherlands.4 In an attempt to stimulate similar activities in hospitals in
Anne Sales, PhD, formerly Research Assistant, Institute for Health Services Research, School of Public Health, University of Minnesota, Minneapolis, is Investigator, HSR&D, VA Puget Sound Health Care System, Seattle. Ira Moscovice, PhD, is Professor and Director, Rural Health Research Center, Division of Health Services Research and Policy, School of Public Health University of Minnesota. Nicole Lurie, MD, MSPH, formerly Associate Professor, Institute for Health Services Research, is Principal Deputy Assistant Secretary for Health, Department of Health and Human Services, Washington, DC.
Support for this article was provided by the Robert Wood Johnson Foundation under Grant No. 13644. Support was also provided by the VA Health Services Research and Development Service. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. Please address requests for correspondence to Anne Sales, PhD, Investigator, HSR&D, VA Puget Sound Health Care System, 1660 S Columbian Way, Seattle, WA 98108; phone 206/764-2068; fax 206/764-2935; e-mail [email protected].
Copyright © 2000 by the Joint Commission on Accreditation of Healthcare Organizations
476
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Article-at-a-Glance Background: Quality improvement approaches such as continuous quality improvement (CQI) and total quality management are widely used, but little is known about how much it costs to use the principles and techniques required to implement CQI processes. In the Robert Wood Johnson Foundation’s Improving the Quality of Hospital Care (IQHC) program, four consortia of hospitals were funded in the early 1990s. Interviews with quality managers at 38 of the consortia hospitals were conducted in 1995 to determine the costs of conducting CQI projects to allow an estimation of the marginal cost of using CQI processes (particularly cross-disciplinary teams) to improve quality of care. CQI projects: Quality managers described 69% of project outcomes as critical to clinical services. Team members identified the issues their teams addressed and selected the project 64% of the time, the methods
the United States, RWJF launched its Improving the Quality of Hospital Care (IQHC) program. The planning phase of this program lasted from 1989 to 1991. Four consortia of hospitals were funded for implementation of the program in late 1991 or early 1992. The four consortia were in Vermont, Iowa, western mountain states (predominantly Utah, where the consortium office was located), and California. Each consortium had a different focus and approach, and each had different memberships, both at inception and throughout the life of the IQHC program. However, one of the activities in common among all four consortia was training and technical assistance to member hospitals in CQI tools and techniques. As part of the implementation of the program, the foundation commissioned an evaluation, which was conducted from 1992 through 1996. The evaluation was split into two phases, one focusing on the earlier components of the program and issues related to setting up the consortia. A second phase evaluated the effectiveness of the programs of each consortium. Two articles have reported on elements of the first phase of the evaluation.5,6 This article describes elements of the second phase of the IQHC program evaluation, in which we focused on effectiveness of consortia programs. Because CQI training and support was one of the few common programs offered by all four consortia, it was
VOLUME
26
NUMBER
of analysis 87% of the time, and the approaches to resolving the problem or issue 97% of the time. Most of the respondents agreed that the team members had the authority to resolve the problem without appealing to higher levels of management. Costs for hospitals’ most recently completed projects varied widely, from $148 for the entire project to $18,590. The length or duration of the projects also varied widely, from 1 month to 66 months. Discussion: In the hospitals included in this sample, all of which were highly self-selected (evidenced by their participation in a voluntary consortium of hospitals focused on quality of care), knowledge of CQI processes appeared to be fairly thorough. Teams appeared to have a reasonable amount of autonomy. New CQI projects should be subjected to scrutiny in terms of their likely contribution to quality of care, as distinct from other positive outcomes.
one of the programs evaluated in the second phase of the evaluation. Our primary purpose is to describe the CQI projects implemented by hospitals participating in consortia, both in terms of their stated aims and in terms of the characteristics of the CQI processes used in conducting the projects. We used criteria established by the Baldrige Commission7 to measure the degree to which the teams followed standards of CQI. We collected data on the costs of conducting the projects so that we could estimate the marginal cost of using CQI processes (particularly cross-disciplinary teams) to improve quality of care.
Background There is a large literature on CQI in health care organizations. Much of the early literature is prescriptive rather than evaluative, describing how to implement principles of CQI. Much of the evaluative literature consists of case studies that describe the results of a single CQI project and its outcomes. This stems from the project-oriented nature of CQI, in which attention is focused on each completed project rather than on CQI itself as a technique or an intervention. More recently, descriptions of a large-scale (16-hospital) study designed to evaluate the impact of CQI processes on care of patients
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
477
THE JOINT COMMISSION undergoing coronary artery bypass graft (CABG) surgery have been published.7 One large-scale study of several hospital systems, which included 61 hospitals, has examined the degree to which hospitals have actually implemented the principles of CQI,8 but it did not include information about specific CQI projects or details of how those projects were accomplished. The investigators concluded that although most of the hospitals included in the study claimed to have implemented CQI principles, there was evidence that these principles had not fully permeated the organizations. Only about onehalf of the hospitals had fully implemented CQI as a method of quality management. In a related article, Shortell and his coauthors9 found that hospitals that exhibited a more participative, flexible structure and were more likely to take risks as organizations were also more likely to have implemented CQI processes throughout the hospital. A large study of 228 CQI projects completed by Veterans Affairs (VA) hospitals in the western United States was conducted in the mid-1990s.10 The study’s primary goal was to develop a model to explain the formation of CQI teams in hospitals. The authors examined team size, composition, age, and a proxy measure for QI, as well as several hospital-level variables. They estimated a regression equation to examine the influence of hospital-level variables on the number of teams formed and the degree to which they affected QI. Although they did not report data on costs of CQI teams or projects, their registry of CQI projects allowed them to gather information on a large number of projects in each of the participating hospitals. In a recent study of 49 separate projects conducted in three units of the same institution,11 the authors found that 60% of the projects had positive outcomes and that introduction of CQI processes had improved morale and satisfaction for both patients and providers. They concluded that the generally positive outcomes and improved satisfaction had been achieved without negative impact on the organization’s budget; however, they did not study the costs associated with the teams or projects. The study by Shortell, Dranove, and others8,12 examining the impact of CQI and organizational culture on outcomes of care for patients undergoing CABG describes effects of both CQI implementation and some aspects of organizational culture on a
478
number of outcomes: risk-adjusted adverse outcomes, functional health status, patient satisfaction, and costs. Findings on clinical outcomes and satisfaction were somewhat mixed; satisfaction with care was higher in hospitals with high CQI scores but longer lengths of stay. Overall, however, CQI was found to have little impact. Costs, which were reported in a separate article,8 were found to vary widely across facilities, and team meetings used for CQI processes were a large contributor to the costs of QI activities. However, the authors found no significant relationship between costs of the QI activities and costs of providing care to patients receiving CABG.
CQI as Process and Technology CQI processes can be treated as a technology for quality management. The elements generally regarded as essential are system and process focus; emphasis on structured problem solving using scientific methods (inductive reasoning and statistical tools); formation of a cross-disciplinary or cross-functional team to address a problem that has been identified; empowerment of team members; and explicit identification of customers who benefit from the process being addressed.8,9 CQI processes are most likely to generate costs in two ways: through identification of problems that might not otherwise be identified and formation of teams. These costs might not be generated using other methods of quality management, since a team would not be formed or a particular problem of importance to one group in an organization might not be deemed sufficiently important to address. Problem identification is one way in which CQI processes are likely to generate costs. Using CQI may lead to identifying more problems than might have been examined without CQI processes in quality management.13 This is analogous to the costs associated with increased case finding in health care generally, such as in preventive screening for health problems. Although it can be argued that finding and fixing problems related to quality of care is better in the long run than not addressing such problems, the short-run costs associated with this approach may, in fact, be relatively large, especially for small or finan-
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Table 1. Characteristics of the Four Consortia
Consortium
Vermont
Number of hospital members in 1992 Number of hospital members in 1995 % of hospitals participating in CQI training by 1995 Year CQI training first offered Response rate to survey* (N)
14 16 81% 1991 92.8% (13)
Iowa
8 48 25% 1994 75.0% (9)
Utah
8 8 75% 1993 100% (6)
California
12 14 71% 1993 100% (10)
* All hospitals participating in the CQI training and support offered by each consortium were included in the survey, whether they were members of the consortium or not. CQI, continuous quality improvement.
cially precarious organizations. Differing perceptions of problems and their relative importance, which may be conditioned on disciplinary perspective,6 can lead to an increase in political processes within organizations as factions vie for attention to the problems they consider important. It can also be argued that other methods of quality management are as effective at identifying problems as CQI. One of the unique features of CQI, however, that might lead to an increase in problem identification is the deliberate inclusion of team members from different disciplines. Because of the particular lenses that different disciplines bring to viewing situations, including those perspectives in the process of identifying problems is likely to increase the number of problems identified. Team formation may also generate costs that might be avoided using other processes to manage quality of care. Because CQI processes require that teams be cross-functional or cross-disciplinary, they are usually composed of at least three people, and often more. The costs generated by using teams to address quality of care problems come from direct costs of personnel, indirect costs in terms of potential opportunity cost (participants might have been occupied doing other things that might have been productive in different ways), and direct costs of supplies, equipment, or other resources required by the team to address the problem. Another indirect way in which team processes may contribute to cost is that teams require time to coalesce and become effective. Having new teams for each identified problem or quality of care issue may lead to long start-up costs or repetition of activities as new teams discover approaches that have been used in the past. However, there is no way of knowing to what other uses these resources might be put if CQI processes were not used for QI. Hospitals might choose to spend the money in other ways
VOLUME
26
NUMBER
related to quality management or might use the resources for entirely different purposes. There are also potential benefits to the use of CQI which are difficult to capture through survey research. One of the most frequently cited positive effects is improvement in staff morale and interest in work. Other ways in which CQI may generate costs in hospitals are through the costs of training employees in CQI tools and techniques, support for quality managers or directors and quality management departments, and other less direct costs borne by the institution in order to use CQI processes. Some of the same hospitals included in this study were part of the sample for the previous research, in which we found wide variation among sample hospitals in whether they had a specific department devoted to quality management (about 30% did not); and if they did, the departmental budget (ranging from $50,000 to $500,000 annually) and the extent of staff training in CQI tools and techniques and associated costs. For those hospitals that trained a large proportion of staff (approximately 40%), costs of such training ranged from about $10,000 to more than $100,000 for at least a one-time training.
CQI Implementation CQI Training and Support Services Across Consortia A total of 41 hospitals received CQI training and support from the four consortia. Not all hospitals that were members of each consortium at the time of the second phase of the evaluation participated in CQI training. The proportion of hospitals in each consortium that were members receiving CQI training and support varied. Characteristics of the consortia and membership participating in CQI training and support are shown in Table 1 (above). The membership of
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
479
THE JOINT COMMISSION Table 2. Characteristics of Hospitals Participating in CQI Training (1995; N=38)*
Consortium Average number of acute beds (SD) Average total budget (SD) % Rural % Joint Commission accredited % Publicly owned % Private, not-for-profit Average length of membership in consortium as of December 1995
Vermont (13)
Iowa (9)
125 (149) $23.6M (13.9M) 87.5% 100% 0 100%
137 (120) $39.7M (10.8M) 66.7% 78% 33.3% 55.6%
4 years
2.25 years
Utah (6)
California (10)
36.7 (18.1) 274 (353) $5.0M (6.9M) $269.3M (472.7M) 83.3% 30.0% 50% 100% 50% 100% 16.7% 0 2.33 years
3.5 years
* SD, standard deviation; CQI, continuous quality improvement.
each consortium varied widely, in number of members, average size of member hospitals, type of hospital in the consortium, and average length of consortium participation. In some cases, nonmember hospitals participated in CQI training and support. Three of the four consortia charged hospitals for this service, with some price differential for member hospitals. Member hospitals offered various reasons for why they chose to participate in CQI training and support. The actual training and support services offered by each consortium also varied, particularly in terms of the frequency of training sessions and the extent of ongoing support after training. Characteristics of the hospitals participating in the CQI training and support, by consortium, are shown in Table 2 (above). Sample According to consortium staff, 44 hospitals received CQI training and support services from the four consortia. In two hospitals, the quality managers did not feel that their hospitals received much training by the consortia; one hospital was very small and did not use CQI processes to any great degree, while the other was a relatively recent member of the consortium and had received its CQI training before joining the consortium. One of the hospitals in the Utah consortium had never completed a CQI project, despite having received training, and the quality manager did not believe that the hospital would be able to use CQI processes in the future. These three hospitals were excluded from the sample. The final sample included 38 of the 41 hospitals, a response rate of 93%. However, not all the hospitals were able to provide data on costs or
480
duration of the projects. A total of 31 usable responses were received on costs of the projects (76% of all respondents). Table 3 (p 481) summarizes the findings in terms of prior CQI training, team composition, and degree to which the team processes conformed with CQI tenets of employee empowerment. Telephone Survey We conducted a telephone survey between September and December 1995 with the quality managers at each of the 41 hospitals in all four consortia that had participated in CQI training and support activities. We believed that they were likely to have a broad picture of quality of care activities throughout the organization. We designed the questionnaire expressly for this study, drawing on the literature on CQI processes and techniques and on elements that had been used in a previous study.7,9 We sought information on the organization’s last completed CQI project, including the project’s title; whether a team had been formed to address the problem; the composition and size of the team, including disciplinary affiliations of each team member; the way in which the problem had been identified; whether the team members had received recognition and/or commendations for participating in the project; statistical tools used in the project; and the purpose or intent of any statistical analysis. We also collected information about some general outcomes of the projects, including the quality managers’ perceptions of how widespread the effects
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Table 3. Prior CQI Experience, Team Composition, and Employee Empowerment, by Consortium*
Vermont % Hospitals with prior training in CQI % With extensive experience in CQI prior to consortium training Average number of members in team† (SD) Average number of administrative staff on team (SD) Average number of physicians on team (SD) Average number of nurses on team (SD) Average number of lab staff on team (SD) Average number of quality managers on team (SD) Average number of other staff on team (SD) % Reporting that teams selected problem % Reporting that teams selected method of analysis % Agreeing that teams had authority to make changes
61
Iowa 55
61.6 22.2 8.0 (3.3) 6.7 (3.1) 1.6 (1.7) 1.4 (1.3) 1.5 (1.2) 0.67 (1.3) 1.6 (1.6) 1.2 (1.8) 0.31 (0.48) 0.11 (0.33) 0.54 (0.52) 0.33 (0.50) 2.6 (1.9) 2.9 (1.4) 69.2 44.4 92.3 87.5 92.3 100.0
Utah 50 33.3 8.7 (7.3) 1.7 (1.5) 0.33 (0.52) 1.8 (1.9) 1.3 (1.9) 0.33 (0.52) 2.7 (1.7) 83.3 100.0 83.3
California 25 20.0 6.6 (3.1) 0.50 (0.85) 1.7 (1.2) 2.3 (1.8) 0.50 (1.1) 0.80 (0.92) 1.9 (1.6) 62.5 75.0 100.0
* CQI, continuous quality improvement; SD, standard deviation. † The
remaining measures describe team composition and employee empowerment for the last completed project only.
of the projects were in the hospital and the degree to which the projects achieved the desired outcomes. We were not able to structure the interviews to collect data about the specific effectiveness of each project. The telephone survey, which was conducted by trained telephone interviewers, lasted approximately 40 to 50 minutes. We developed a separate questionnaire on costs of the CQI projects on the basis of the existing literature and pilot tested them in three local hospitals before fielding them in the survey. In this questionnaire, which we also administered to the quality managers between September and December 1995, we collected data on the titles of the members of each project team; the amount of time each member of the team had spent on the project; the wage rates (estimated if necessary) for each type of team member; and costs of any other resources used in the project. Wage rates were estimated by the quality manager responding to the survey. We validated some of the wage rates using data from national studies and found that they were consistent with other sources for the geographic region. We calculated personnel time costs by multiplying the hours each team member spent on project team activities (estimated by the quality manager) by the hourly wage rate for that team member.
VOLUME
26
NUMBER
These cost data were collected either by telephone interview or by fax. In all cases, the questions included in the cost data were sent ahead of the telephone interview to the quality managers, as part of the letter explaining the study and requesting participation. Some of these were returned before the telephone survey, some were completed during the telephone survey, and some were returned after completion of the telephone survey. In a small number of cases, a second telephone call was made to complete the cost component of the study. In addition to the telephone surveys of hospital quality managers, between June and November 1995 we also conducted site visits to each consortium and talked to the staff who were responsible for the CQI training and support services.
CQI Projects Experience The 38 hospitals’ experiences in using CQI varied by consortia—from those that had completed only one pilot project to those that had been using CQI processes for several years and had many completed projects. Whether or not hospitals had been using CQI prior to receiving training from the consortium also varied by consortium. Yet the small sample size resulted in insufficient power to detect statistically significant differences across consortia.
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
481
THE JOINT COMMISSION Team Participation All 38 hospitals in the final sample—those that had used consortium services for CQI training and had completed a pilot project—had formed crossdisciplinary teams for the project they were describing. Teams ranged in size from 3 members to 23; at least two disciplines were represented on each team. Mean team size was 7.4 (standard deviation [SD], 3.99), with a median of 6.5. Team composition by discipline was highly variable (the quality managers made the disciplinary classifications, choosing from the categories presented). Quality managers participated in 44.7% of the teams, laboratory staff in 29.0%, administrators in 55.3%, physicians in 60.5%, and nurses in 65.8%. Staff in the “other” category—including personnel such as radiology technicians, dietitians, respiratory therapists, chaplains, and volunteer coordinators—participated in 89.5%. Employee empowerment is one of the five elements of CQI processes that we attempted to measure. We assessed this dimension using a number of items that were borrowed from previous work assessing degree of implementation of CQI in hospitals.9 Team members identified the issue their team addressed and selected the project 64% of the time, the methods of analysis 87% of the time, and the approaches to resolving the problem or issue 97% of the time. Twenty-seven (75%) of the 38 respondents agreed with the statement that team members had the authority to resolve the problem without appealing to higher levels of management. This appears to be a relatively high level of employee empowerment and involvement in decision making perceived by quality managers. Types of Projects CQI projects, listed in Table 4 (p 483), were classified on the basis of the quality manager’s judgment as to whether they were primarily clinical or nonclinical in nature. We also asked quality managers to rate each project on a five-point scale in terms of how critical the outcomes of the project had been, first for clinical services in the hospital, and second for nonclinical services. In only 3 of the 38 projects did the quality managers consider the problems to be unresolved (and the projects therefore ineffective)—too small an n, we
482
believed, to enable us to explore differences between effective and ineffective projects. In terms of the ratings of the importance of the projects’ outcomes, we collapsed each question into a dichotomous outcome (not critical/critical) and then combined the two into a single variable, in which we counted the outcome of the project as critical to the hospital if it was deemed critical either to clinical or to nonclinical services. Quality managers described 69% of project outcomes as critical to clinical services. The projects varied considerably in topic. Some of the projects (for example, one focusing on a transitional care program) dealt with a specific program within a hospital. Others dealt with hospitalwide issues, such as paperwork and documentation requirements across the entire facility. A few focused on a specific health problem or service, such as pressure ulcers or epidural anesthesia. Most of the projects had some direct relationship to patient care, although a few (such as the redesign of an award and employee recognition project) focused on issues that have little or no direct effect on patient care. We examined bivariate correlations between several hospital characteristics and whether a project was considered primarily clinical by the quality manager. We found that a hospital being private is directly correlated with the project being considered primarily clinical; the status of CQI implementation three years previously, use of CQI prior to training by the consortium, and amount of progress on implementing CQI processes were not significantly correlated with whether the project was considered primarily clinical. Costs and Duration of Projects Costs for the CQI projects varied widely, from $148 for the entire project to $18,590 (mean, $5,047 [SD, $4,680]; median, $3,580). The length or duration of the projects also varied widely, from 1 month to 66 months (mean, 10.7 months [SD, 11.9]; median, 8 months). It was difficult to compare costs for projects without using project duration as a denominator. We also calculated a measure of the labor intensity of the project, defined as the total personnel hours divided by the number of months the project lasted (person hours/month). The mean and median total costs, duration, cost per month, and
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Table 4. Projects Included in the Study (N = 38)*
Project considered by quality manager to have critical outcome for either clinical or nonclinical services
Project title Project is considered primarily clinical Early discharge program for normal vaginal delivery Paper chase Pathology report turnaround time Identification and treatment of patients with chest pain in emergency room Patient assessment Chest pain protocol Epidural anesthesia Point-of-care glucose testing Document turnaround time Efficiency of medical staff sections Same-day surgery process Patient transport team Food delivery project Weight team Pressure ulcers Home health discharge planning Ventilator management Unavailability of diagnostic imaging films Timely reporting of panic lab values Conscious sedation
Yes Yes Yes Yes No Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes
Project is considered primarily nonclinical Managing emergency room transfers to other facilities Medical records task force Redesign award and employee recognition Transitional care program Ineffective billing process Interagency patient transfer Chat book Timeliness of ancillary tests in emergency room Paper flow pit Patient satisfaction Receiving and shipping Outpatient clinic wait time Lab consolidation Engineering work order team Blood wastage Antibiotics on time Reduction in time to administration of thrombolytic therapy in emergency room Operating room turnaround time
Yes Yes Yes Yes Yes Yes No No Yes Yes Yes Yes Yes No No No No No
* Project titles in italics indicate problems considered unresolved by the quality manager.
VOLUME
26
NUMBER
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
483
THE JOINT COMMISSION Table 5. Costs, Duration, Costs per Month, and Labor Intensity of CQI Projects, by Consortium*
Consortium Vermont: Mean (SE) Median Iowa: Mean (SE) Median Utah: Mean (SE) Median California: Mean (SE) Median
Total cost
Duration in months
Cost per month
Labor intensity (total FTEE person hours/month)
$4,430 (1,139) $2,700 $4,902 (1,589) $3,980 $2,898 (746) $3,580 $6,562 (2,125) $5,190
8.0 (1.2) 7.0 19.1 (7.3) 15.5 7.2 (3.1) 3.0 8.8 (1.7) 10.0
$931 (311) $465 $543 (285) $261 $809 (413) $210 $1,112 (488) $969
33.9 (13.7) 10.7 24.0 (10.4) 14.3 36.2 (15.2) 13.5 32.1 (11.6) 25.0
* FTEE, full-time equivalent employee; SE, standard error of the mean.
intensity of the projects in each consortium are shown in Table 5 (above). The CQI projects varied considerably across sites in total costs, monthly cost, and labor intensity, although none of these differences was statistically significant. In addition to exploring the differences in cost, duration, and intensity of the projects by consortium, we examined differences in these elements between projects that quality managers considered to have critical outcomes either for clinical or nonclinical services in the hospitals. While there are very large differences between the means for these two groups of projects, none of the differences were significant, again because of the very wide variation in the distributions of the variables in both groups (Table 6, p 486). As in the previous discussion, the sample size in this study is too small to detect statistically significant differences. However, it appears that the observed higher costs in the group of projects deemed to have critical outcomes might be related to the higher intensity (person hours/month), as well as longer duration of the projects.
Discussion We have presented a detailed picture of the costs and use of CQI processes among a set of CQI projects, sampled from all the projects completed and under way in 38 hospitals involved in the IQHC program. Lack of adequate power because of small sample size was an important limitation of this study.
484
Sample size considerations might limit the likelihood of further research in this area, but it is important to consider the issue. Quantitative analysis in CQI is rife with small-sample problems and low underlying rates. For example, low-denominator numbers (such as the number of diabetics in a given practice or hospital to whom a specific intervention applies, or as in mortality rates, which makes it difficult to use mortality as an outcome in a small-area study) make it important to think carefully about appropriate tests of statistical significance, as Diehr and colleagues have pointed out with respect to small-area analysis.14–16 Given the nature of CQI processes and their focus on individual projects, it is difficult to accumulate a sufficient number of observations to enable statistical inference.* There are some interesting lessons to learn from the data collected for this study. The first is that in the hospitals included in this sample, all of which were highly self-selected (evidenced by their participation * Because this is the first study to report costs of team processes for individual CQI projects, it is instructive to calculate the sample sizes needed to achieve a probability of statistical significance in the difference between mean costs for CQI projects in one setting compared to another. The difference that will be used in this example is the difference between the mean cost of CQI projects in Vermont compared to those in Iowa: Mean costs in Vermont are $4,430 (SD, $1,139), and mean costs in Iowa are $4,902 (SD, $1,589). The difference in the means is $472. With power set at .90 and alpha at .05 for a two-sided test of statistical significance, the minimum number of CQI projects in each sample required to detect the observed mean difference would be 181. We included 31 projects in our sample, each from a different hospital. If projects from the same hospital were included, it might be necessary to correct estimates for cluster sampling, requiring an even larger sample size.
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
in a voluntary consortium of hospitals focused on quality of care), knowledge of CQI processes appears to be fairly thorough, at least from the perspective of quality managers.* Adherence to five basic tenets of CQI, described at the beginning of this article, is evident in most of the projects described. Many of the hospitals, by the time we conducted this survey, were fairly advanced in their implementation of projects based on CQI. The use of cross-disciplinary teams appears to be fostering collaboration among different disciplines in hospitals, one of the primary goals of CQI. Teams appear to have a reasonable amount of autonomy, at least in the view of quality managers in the hospitals. It is interesting to note that some of the core disciplines (nurses and physicians) providing health care in hospitals did not appear to be represented in large numbers on CQI teams (at 18% and 24% of total team membership, respectively), whereas administrators and other staff were relatively well represented (15% and 35%, respectively). Physician participation in CQI projects is often considered difficult to achieve,17 yet when physicians were employees (in this study, primarily in the California consortium), rates of physician participation were higher than when they were predominantly either self-employed or employed by an organization other than the hospital. Participation by hospital employees was generally considered part of the usual work schedule. Physicians tended to spend the least amount of time in project work, while other types of staff generally participated equal amounts of time. This probably represents the attempts of hospital staff to minimize the time costs for physicians of CQI team work and is probably one of the balancing acts necessary to gain their participation in an activity that is valued by the organization. In terms of costs of projects, however, limiting physician involvement and time commitment is a way of keeping project and team costs lower than they would be if physicians spent a great deal of time on the
projects. Physician hourly rates were among the highest wage rates of any discipline involved in projects.† Administrators were next highest in reported wage rates, and, like physicians, tended to spend less time on projects than other team members. Although the average costs of the team work needed for CQI project completion do not seem extremely high, it is important to remember that most hospitals conduct several CQI projects a year, and especially in large hospitals, have multiple CQI projects going on at one time. If the average costs reported in this article were multiplied by five to ten projects a year for a hospital, the costs of team work needed to conduct CQI projects begins to appear higher. In addition, it is worth bearing in mind that there is usually a great deal of additional work not related to CQI projects being carried out by quality management staff, which may include monitoring various indicators of quality of care such as nosocomial infection rates, and conducting required assessments and reviews of systems required for Joint Commission accreditation. Managing quality of care is a complicated business for most moderate and large hospitals and health care organizations, even without CQI processes and their demands. Adding costs of CQI teams to “routine” quality management or quality assurance may, for hospitals facing severe budget pressures, prompt examination of the value of quality management activities as a whole. In the mid-1990s, at least in some public hospitals in California, the results of such an examination were to eliminate the quality management staff. Although CQI processes have only relatively recently been adopted by health care organizations, they have come to be regarded as staples of quality management in most hospitals in the United States. This is in part because of mandates and perceived mandates by accrediting bodies, and in part because many hospital employees who receive training in the processes and techniques of CQI become very enthusiastic about its potential to help in dealing with problems in quality of care.
* We did not examine the extent to which staff throughout the hospital had received training or participated in CQI activities. In previous research, which included some of the same hospitals as this sample, we found that in about 30% of hospitals surveyed, more than 50% of the staff had received some type of CQI training.
† We collected information about physician hourly rates whether the hospital paid for the physician’s time or not, since we were interested in assessing as much of the cost of CQI projects as possible. Costs for the projects were mostly driven by hourly rates and time spent on team meetings.
VOLUME
26
NUMBER
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
485
THE JOINT COMMISSION Table 6. Costs, Duration, Costs per Month, and Labor Intensity of CQI Projects by Whether the Project Outcome Was Considered Critical*
Program outcome considered critical? Projects considered critical: Mean (SE) Median Projects not considered critical: Mean (SE) Median
Total cost
Duration in months
Cost per month
Labor intensity (total FTEE person hours/month)
$5,114 (4,413) $4,166
11.18 (12.72) 10.76
$868 (1,069) $294
17.43 (13.1) 11.9
$4,854 (5,671) $2,675
8.83 (2.99) 9.0
$494 (424) $234
24.0 (10.4) 14.3
* CQI, continuous quality improvement; FTEE, full-time equivalent employee; SE, standard error of the mean.
However, the adoption of CQI in hospitals has received relatively little systematic study to date. This is due, in part, to the difficulty of studying these processes in ways that permit comparative quantitative analysis. Our findings indicate that a large number of projects would need to be included in a comparative study to fully examine the factors that influence the costs of CQI projects. In addition, further research on whether CQI projects ultimately affect patient outcomes, and whether they are cost-effective compared to other methods of managing quality of care, is required. Several approaches to designing studies of CQI cost-effectiveness could be developed. One might be to design a study that collects data on several CQI projects for each hospital in a representative sample of hospitals that have been completed within, for example, the past two years. The number of CQI projects that we estimate would be needed to detect statistically significant differences in costs of projects, based on our findings, is about 180. Although it would not be difficult to accumulate this number of projects in a hospital sample similar to the one in this study (38–40 hospitals), it would be necessary to perform cluster correction to ensure that statistical inferences appropriately adjust for including more than one study per hospital in the sample. Using outcomes from a cluster sample such as this to evaluate consortia, like the ones in our study, would probably require more than the 180 or so projects we suggest would be necessary if the sample of projects came from different hospitals. CQI is one model for managing and improving quality of care in health care organizations. Alternative and sometimes complementary models include performance management, in which the focus is on outcomes
486
rather than process, and more “traditional” methods of quality assurance, including a focus on indicators (such as nosocomial infection rates) of quality of care. While none of these models provide a final solution to how quality of care can be managed and ultimately improved, we should regard these as technologies to be assessed by the same criteria (such as cost-effectiveness) used to assess other technologies. In an era of cost containment in health care, questions related to the costs of doing business must be addressed. While our study cannot provide definitive answers to questions about how effective or cost-effective CQI processes are in hospitals, some of our findings suggest approaches that might be useful to quality managers. First, new CQI projects should be subjected to scrutiny in terms of their likely contribution to quality of care, as distinct from other positive outcomes (for example, improving efficiency in nonclinical services, enhancing the public image of the hospital or health care facility). If the primary goal is improved direct quality of care, then the project should have goals and objectives that foster that as an outcome. The composition of the team should also be carefully assessed. Because of heavy work loads and demands on time by all staff in a hospital, it is often tempting to include people who might not have direct knowledge of the processes to be improved as part of a CQI team, to provide additional human resources. While this is sometimes helpful, it should be recognized that it may extend the time required to complete the project, and the additional costs associated with longer duration of a project should be taken into account as teams are formed and their objectives are determined.
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Commitment from team members is a critical element in timely completion of a project, and if key participants are not able to make a commitment to a project, there should be careful review of whether the project should be chartered. CQI tools and techniques have largely been embraced in health care, despite the fact that to date, relatively little comparative empirical research exists on the effectiveness of these techniques.18 More recent studies that have been published8,12 find mixed results, indicating that it is difficult to demonstrate that CQI as a dominant method of managing QI is
both effective and cost-effective. However, since few of the other methods of quality management have received the same scrutiny, it is difficult to make the necessary judgments as to how best to improve or manage the quality of care in health care settings. During the past decade, we have improved our capacity to measure quality of care and its outcomes. We need to apply this improved capacity to examining the technologies we use to manage quality of care, measuring the effectiveness of different approaches, and exploring the relative costs of different ways of managing quality. J
References 1. Berwick D: Continuous improvement as an ideal in health care. N Engl J Med 320:53–56, 1989. 2. Laffel G, Blumenthal D: The case for using industrial quality management science in health care organizations. JAMA 262:2869– 2873, 1989. 3. O’Leary DS: Accreditation in the quality improvement mold: A vision for tomorrow. Qual Rev Bull 17:72–77, 1991. 4. Kabcenell AI, Cohen AB, Merrill JC: Importing a model of hospital quality from the Netherlands. Health Aff 10:240–245, 1991. 5. Sales A, et al: Is quality in the eye of the beholder? Jt Comm J Qual Improv 21:219–225, 1995. 6. Sales A, Moscovice I, Lurie N:
VOLUME
26
NUMBER
Measuring seriousness of hospital quality of care issues. Jt Comm J Qual Improv 22:811–816, 1996. 7. Dranove D, et al: The cost of efforts to improve quality. Med Care 37:1084–1087, 1999. 8. Shortell SM, et al: Assessing the evidence on CQI: Is the glass half empty or half full? Hosp Health Serv Admin 40(1):4–24, 1995. 9. Shortell SM, et al: Assessing the impact of continuous quality improvement/total quality management: Concept versus implementation. Health Serv Res 30:377–401, 1995. 10. Lammers JC, et al: Total quality management in hospitals: The contributions of commitment, quality councils, teams, budgets, and training to perceived improve-
ment at Veterans Health Administration hospitals. Med Care 34:463– 478, 1996. 11. Ziegenfuss JT, et al: Engineering quality through organization change: A study of patient care initiatives by teams. Am J Med Qual 13(1):44–51, 1998. 12. Shortell SM, et al: Assessing the impact of total quality management and organizational culture on multiple outcomes of care for coronary artery bypass graft surgery patients. Med Care 38:207–217, 2000. 13. Traska MR: Don’t try total quality improvement if you think it will solve problems it shouldn’t. Healthcare Competition Week 9(6): 9–10, 1992. 14. Diehr P, Grembowski D: A
small area simulation approach to determining excess variation in dental procedure rates. Am J Public Health 80:1343–1348, 1990. 15. Cain KC, Diehr P: Testing the null hypothesis in small area analysis. Health Serv Res 27:267– 294, 1992. 16. Diehr P, et al: 1992. Can small-area analysis detect variation in surgery rates? The power of small-area variation analysis. Med Care 30:484–502, 1992. 17. Shortell SM, et al: Remaking Health Care in America: Building Organized Delivery Systems. San Francisco: Jossey-Bass, 1996. 18. Bigelow B, Arndt M: Total quality management: Field of dreams? Health Care Manage Rev 20(4):15–25, 1995.
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
487
Implementing CQI Projects in Hospitals ANNE SALES, PHD IRA MOSCOVICE, PHD NICOLE LURIE, MD, MSPH
he application of quality improvement (QI) approaches such as continuous quality improvement (CQI) and total quality management (TQM) to health care organizations has been a relatively recent phenomenon.1,2 Quality assurance approaches, still dominant in health care organizations in the United States during the 1980s, were often characterized as highly punitive, with an emphasis on identifying individuals who were thought to cause problems in quality of care rather than attempting to find sources of problems in the systems of care. CQI (or TQM) advocates suggested that these approaches offered a more effective way to approach problems in quality of care and a more humane method of dealing with health care providers. CQI has been a mainstay of quality management in health
T
care organizations since the early 1990s, when the tools and techniques of CQI were strongly encouraged as a component for accreditation by the Joint Commission on Accreditation of Healthcare Organizations (Oakbrook Terrace, Ill).3 Concurrent with this relatively new approach to managing quality of care in hospitals in the United States, in 1989 the Robert Wood Johnson Foundation (RWJF; Princeton, NJ) began a multiyear program aimed at helping hospitals grapple with issues related to quality of care. The intent of the program was to facilitate new models and methods of responding to quality management, and a prime motivator for the program was a model of hospital consortia organized around quality of care issues in the Netherlands.4 In an attempt to stimulate similar activities in hospitals in
Anne Sales, PhD, formerly Research Assistant, Institute for Health Services Research, School of Public Health, University of Minnesota, Minneapolis, is Investigator, HSR&D, VA Puget Sound Health Care System, Seattle. Ira Moscovice, PhD, is Professor and Director, Rural Health Research Center, Division of Health Services Research and Policy, School of Public Health University of Minnesota. Nicole Lurie, MD, MSPH, formerly Associate Professor, Institute for Health Services Research, is Principal Deputy Assistant Secretary for Health, Department of Health and Human Services, Washington, DC.
Support for this article was provided by the Robert Wood Johnson Foundation under Grant No. 13644. Support was also provided by the VA Health Services Research and Development Service. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. Please address requests for correspondence to Anne Sales, PhD, Investigator, HSR&D, VA Puget Sound Health Care System, 1660 S Columbian Way, Seattle, WA 98108; phone 206/764-2068; fax 206/764-2935; e-mail [email protected].
Copyright © 2000 by the Joint Commission on Accreditation of Healthcare Organizations
476
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Article-at-a-Glance Background: Quality improvement approaches such as continuous quality improvement (CQI) and total quality management are widely used, but little is known about how much it costs to use the principles and techniques required to implement CQI processes. In the Robert Wood Johnson Foundation’s Improving the Quality of Hospital Care (IQHC) program, four consortia of hospitals were funded in the early 1990s. Interviews with quality managers at 38 of the consortia hospitals were conducted in 1995 to determine the costs of conducting CQI projects to allow an estimation of the marginal cost of using CQI processes (particularly cross-disciplinary teams) to improve quality of care. CQI projects: Quality managers described 69% of project outcomes as critical to clinical services. Team members identified the issues their teams addressed and selected the project 64% of the time, the methods
the United States, RWJF launched its Improving the Quality of Hospital Care (IQHC) program. The planning phase of this program lasted from 1989 to 1991. Four consortia of hospitals were funded for implementation of the program in late 1991 or early 1992. The four consortia were in Vermont, Iowa, western mountain states (predominantly Utah, where the consortium office was located), and California. Each consortium had a different focus and approach, and each had different memberships, both at inception and throughout the life of the IQHC program. However, one of the activities in common among all four consortia was training and technical assistance to member hospitals in CQI tools and techniques. As part of the implementation of the program, the foundation commissioned an evaluation, which was conducted from 1992 through 1996. The evaluation was split into two phases, one focusing on the earlier components of the program and issues related to setting up the consortia. A second phase evaluated the effectiveness of the programs of each consortium. Two articles have reported on elements of the first phase of the evaluation.5,6 This article describes elements of the second phase of the IQHC program evaluation, in which we focused on effectiveness of consortia programs. Because CQI training and support was one of the few common programs offered by all four consortia, it was
VOLUME
26
NUMBER
of analysis 87% of the time, and the approaches to resolving the problem or issue 97% of the time. Most of the respondents agreed that the team members had the authority to resolve the problem without appealing to higher levels of management. Costs for hospitals’ most recently completed projects varied widely, from $148 for the entire project to $18,590. The length or duration of the projects also varied widely, from 1 month to 66 months. Discussion: In the hospitals included in this sample, all of which were highly self-selected (evidenced by their participation in a voluntary consortium of hospitals focused on quality of care), knowledge of CQI processes appeared to be fairly thorough. Teams appeared to have a reasonable amount of autonomy. New CQI projects should be subjected to scrutiny in terms of their likely contribution to quality of care, as distinct from other positive outcomes.
one of the programs evaluated in the second phase of the evaluation. Our primary purpose is to describe the CQI projects implemented by hospitals participating in consortia, both in terms of their stated aims and in terms of the characteristics of the CQI processes used in conducting the projects. We used criteria established by the Baldrige Commission7 to measure the degree to which the teams followed standards of CQI. We collected data on the costs of conducting the projects so that we could estimate the marginal cost of using CQI processes (particularly cross-disciplinary teams) to improve quality of care.
Background There is a large literature on CQI in health care organizations. Much of the early literature is prescriptive rather than evaluative, describing how to implement principles of CQI. Much of the evaluative literature consists of case studies that describe the results of a single CQI project and its outcomes. This stems from the project-oriented nature of CQI, in which attention is focused on each completed project rather than on CQI itself as a technique or an intervention. More recently, descriptions of a large-scale (16-hospital) study designed to evaluate the impact of CQI processes on care of patients
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
477
THE JOINT COMMISSION undergoing coronary artery bypass graft (CABG) surgery have been published.7 One large-scale study of several hospital systems, which included 61 hospitals, has examined the degree to which hospitals have actually implemented the principles of CQI,8 but it did not include information about specific CQI projects or details of how those projects were accomplished. The investigators concluded that although most of the hospitals included in the study claimed to have implemented CQI principles, there was evidence that these principles had not fully permeated the organizations. Only about onehalf of the hospitals had fully implemented CQI as a method of quality management. In a related article, Shortell and his coauthors9 found that hospitals that exhibited a more participative, flexible structure and were more likely to take risks as organizations were also more likely to have implemented CQI processes throughout the hospital. A large study of 228 CQI projects completed by Veterans Affairs (VA) hospitals in the western United States was conducted in the mid-1990s.10 The study’s primary goal was to develop a model to explain the formation of CQI teams in hospitals. The authors examined team size, composition, age, and a proxy measure for QI, as well as several hospital-level variables. They estimated a regression equation to examine the influence of hospital-level variables on the number of teams formed and the degree to which they affected QI. Although they did not report data on costs of CQI teams or projects, their registry of CQI projects allowed them to gather information on a large number of projects in each of the participating hospitals. In a recent study of 49 separate projects conducted in three units of the same institution,11 the authors found that 60% of the projects had positive outcomes and that introduction of CQI processes had improved morale and satisfaction for both patients and providers. They concluded that the generally positive outcomes and improved satisfaction had been achieved without negative impact on the organization’s budget; however, they did not study the costs associated with the teams or projects. The study by Shortell, Dranove, and others8,12 examining the impact of CQI and organizational culture on outcomes of care for patients undergoing CABG describes effects of both CQI implementation and some aspects of organizational culture on a
478
number of outcomes: risk-adjusted adverse outcomes, functional health status, patient satisfaction, and costs. Findings on clinical outcomes and satisfaction were somewhat mixed; satisfaction with care was higher in hospitals with high CQI scores but longer lengths of stay. Overall, however, CQI was found to have little impact. Costs, which were reported in a separate article,8 were found to vary widely across facilities, and team meetings used for CQI processes were a large contributor to the costs of QI activities. However, the authors found no significant relationship between costs of the QI activities and costs of providing care to patients receiving CABG.
CQI as Process and Technology CQI processes can be treated as a technology for quality management. The elements generally regarded as essential are system and process focus; emphasis on structured problem solving using scientific methods (inductive reasoning and statistical tools); formation of a cross-disciplinary or cross-functional team to address a problem that has been identified; empowerment of team members; and explicit identification of customers who benefit from the process being addressed.8,9 CQI processes are most likely to generate costs in two ways: through identification of problems that might not otherwise be identified and formation of teams. These costs might not be generated using other methods of quality management, since a team would not be formed or a particular problem of importance to one group in an organization might not be deemed sufficiently important to address. Problem identification is one way in which CQI processes are likely to generate costs. Using CQI may lead to identifying more problems than might have been examined without CQI processes in quality management.13 This is analogous to the costs associated with increased case finding in health care generally, such as in preventive screening for health problems. Although it can be argued that finding and fixing problems related to quality of care is better in the long run than not addressing such problems, the short-run costs associated with this approach may, in fact, be relatively large, especially for small or finan-
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Table 1. Characteristics of the Four Consortia
Consortium
Vermont
Number of hospital members in 1992 Number of hospital members in 1995 % of hospitals participating in CQI training by 1995 Year CQI training first offered Response rate to survey* (N)
14 16 81% 1991 92.8% (13)
Iowa
8 48 25% 1994 75.0% (9)
Utah
8 8 75% 1993 100% (6)
California
12 14 71% 1993 100% (10)
* All hospitals participating in the CQI training and support offered by each consortium were included in the survey, whether they were members of the consortium or not. CQI, continuous quality improvement.
cially precarious organizations. Differing perceptions of problems and their relative importance, which may be conditioned on disciplinary perspective,6 can lead to an increase in political processes within organizations as factions vie for attention to the problems they consider important. It can also be argued that other methods of quality management are as effective at identifying problems as CQI. One of the unique features of CQI, however, that might lead to an increase in problem identification is the deliberate inclusion of team members from different disciplines. Because of the particular lenses that different disciplines bring to viewing situations, including those perspectives in the process of identifying problems is likely to increase the number of problems identified. Team formation may also generate costs that might be avoided using other processes to manage quality of care. Because CQI processes require that teams be cross-functional or cross-disciplinary, they are usually composed of at least three people, and often more. The costs generated by using teams to address quality of care problems come from direct costs of personnel, indirect costs in terms of potential opportunity cost (participants might have been occupied doing other things that might have been productive in different ways), and direct costs of supplies, equipment, or other resources required by the team to address the problem. Another indirect way in which team processes may contribute to cost is that teams require time to coalesce and become effective. Having new teams for each identified problem or quality of care issue may lead to long start-up costs or repetition of activities as new teams discover approaches that have been used in the past. However, there is no way of knowing to what other uses these resources might be put if CQI processes were not used for QI. Hospitals might choose to spend the money in other ways
VOLUME
26
NUMBER
related to quality management or might use the resources for entirely different purposes. There are also potential benefits to the use of CQI which are difficult to capture through survey research. One of the most frequently cited positive effects is improvement in staff morale and interest in work. Other ways in which CQI may generate costs in hospitals are through the costs of training employees in CQI tools and techniques, support for quality managers or directors and quality management departments, and other less direct costs borne by the institution in order to use CQI processes. Some of the same hospitals included in this study were part of the sample for the previous research, in which we found wide variation among sample hospitals in whether they had a specific department devoted to quality management (about 30% did not); and if they did, the departmental budget (ranging from $50,000 to $500,000 annually) and the extent of staff training in CQI tools and techniques and associated costs. For those hospitals that trained a large proportion of staff (approximately 40%), costs of such training ranged from about $10,000 to more than $100,000 for at least a one-time training.
CQI Implementation CQI Training and Support Services Across Consortia A total of 41 hospitals received CQI training and support from the four consortia. Not all hospitals that were members of each consortium at the time of the second phase of the evaluation participated in CQI training. The proportion of hospitals in each consortium that were members receiving CQI training and support varied. Characteristics of the consortia and membership participating in CQI training and support are shown in Table 1 (above). The membership of
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
479
THE JOINT COMMISSION Table 2. Characteristics of Hospitals Participating in CQI Training (1995; N=38)*
Consortium Average number of acute beds (SD) Average total budget (SD) % Rural % Joint Commission accredited % Publicly owned % Private, not-for-profit Average length of membership in consortium as of December 1995
Vermont (13)
Iowa (9)
125 (149) $23.6M (13.9M) 87.5% 100% 0 100%
137 (120) $39.7M (10.8M) 66.7% 78% 33.3% 55.6%
4 years
2.25 years
Utah (6)
California (10)
36.7 (18.1) 274 (353) $5.0M (6.9M) $269.3M (472.7M) 83.3% 30.0% 50% 100% 50% 100% 16.7% 0 2.33 years
3.5 years
* SD, standard deviation; CQI, continuous quality improvement.
each consortium varied widely, in number of members, average size of member hospitals, type of hospital in the consortium, and average length of consortium participation. In some cases, nonmember hospitals participated in CQI training and support. Three of the four consortia charged hospitals for this service, with some price differential for member hospitals. Member hospitals offered various reasons for why they chose to participate in CQI training and support. The actual training and support services offered by each consortium also varied, particularly in terms of the frequency of training sessions and the extent of ongoing support after training. Characteristics of the hospitals participating in the CQI training and support, by consortium, are shown in Table 2 (above). Sample According to consortium staff, 44 hospitals received CQI training and support services from the four consortia. In two hospitals, the quality managers did not feel that their hospitals received much training by the consortia; one hospital was very small and did not use CQI processes to any great degree, while the other was a relatively recent member of the consortium and had received its CQI training before joining the consortium. One of the hospitals in the Utah consortium had never completed a CQI project, despite having received training, and the quality manager did not believe that the hospital would be able to use CQI processes in the future. These three hospitals were excluded from the sample. The final sample included 38 of the 41 hospitals, a response rate of 93%. However, not all the hospitals were able to provide data on costs or
480
duration of the projects. A total of 31 usable responses were received on costs of the projects (76% of all respondents). Table 3 (p 481) summarizes the findings in terms of prior CQI training, team composition, and degree to which the team processes conformed with CQI tenets of employee empowerment. Telephone Survey We conducted a telephone survey between September and December 1995 with the quality managers at each of the 41 hospitals in all four consortia that had participated in CQI training and support activities. We believed that they were likely to have a broad picture of quality of care activities throughout the organization. We designed the questionnaire expressly for this study, drawing on the literature on CQI processes and techniques and on elements that had been used in a previous study.7,9 We sought information on the organization’s last completed CQI project, including the project’s title; whether a team had been formed to address the problem; the composition and size of the team, including disciplinary affiliations of each team member; the way in which the problem had been identified; whether the team members had received recognition and/or commendations for participating in the project; statistical tools used in the project; and the purpose or intent of any statistical analysis. We also collected information about some general outcomes of the projects, including the quality managers’ perceptions of how widespread the effects
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Table 3. Prior CQI Experience, Team Composition, and Employee Empowerment, by Consortium*
Vermont % Hospitals with prior training in CQI % With extensive experience in CQI prior to consortium training Average number of members in team† (SD) Average number of administrative staff on team (SD) Average number of physicians on team (SD) Average number of nurses on team (SD) Average number of lab staff on team (SD) Average number of quality managers on team (SD) Average number of other staff on team (SD) % Reporting that teams selected problem % Reporting that teams selected method of analysis % Agreeing that teams had authority to make changes
61
Iowa 55
61.6 22.2 8.0 (3.3) 6.7 (3.1) 1.6 (1.7) 1.4 (1.3) 1.5 (1.2) 0.67 (1.3) 1.6 (1.6) 1.2 (1.8) 0.31 (0.48) 0.11 (0.33) 0.54 (0.52) 0.33 (0.50) 2.6 (1.9) 2.9 (1.4) 69.2 44.4 92.3 87.5 92.3 100.0
Utah 50 33.3 8.7 (7.3) 1.7 (1.5) 0.33 (0.52) 1.8 (1.9) 1.3 (1.9) 0.33 (0.52) 2.7 (1.7) 83.3 100.0 83.3
California 25 20.0 6.6 (3.1) 0.50 (0.85) 1.7 (1.2) 2.3 (1.8) 0.50 (1.1) 0.80 (0.92) 1.9 (1.6) 62.5 75.0 100.0
* CQI, continuous quality improvement; SD, standard deviation. † The
remaining measures describe team composition and employee empowerment for the last completed project only.
of the projects were in the hospital and the degree to which the projects achieved the desired outcomes. We were not able to structure the interviews to collect data about the specific effectiveness of each project. The telephone survey, which was conducted by trained telephone interviewers, lasted approximately 40 to 50 minutes. We developed a separate questionnaire on costs of the CQI projects on the basis of the existing literature and pilot tested them in three local hospitals before fielding them in the survey. In this questionnaire, which we also administered to the quality managers between September and December 1995, we collected data on the titles of the members of each project team; the amount of time each member of the team had spent on the project; the wage rates (estimated if necessary) for each type of team member; and costs of any other resources used in the project. Wage rates were estimated by the quality manager responding to the survey. We validated some of the wage rates using data from national studies and found that they were consistent with other sources for the geographic region. We calculated personnel time costs by multiplying the hours each team member spent on project team activities (estimated by the quality manager) by the hourly wage rate for that team member.
VOLUME
26
NUMBER
These cost data were collected either by telephone interview or by fax. In all cases, the questions included in the cost data were sent ahead of the telephone interview to the quality managers, as part of the letter explaining the study and requesting participation. Some of these were returned before the telephone survey, some were completed during the telephone survey, and some were returned after completion of the telephone survey. In a small number of cases, a second telephone call was made to complete the cost component of the study. In addition to the telephone surveys of hospital quality managers, between June and November 1995 we also conducted site visits to each consortium and talked to the staff who were responsible for the CQI training and support services.
CQI Projects Experience The 38 hospitals’ experiences in using CQI varied by consortia—from those that had completed only one pilot project to those that had been using CQI processes for several years and had many completed projects. Whether or not hospitals had been using CQI prior to receiving training from the consortium also varied by consortium. Yet the small sample size resulted in insufficient power to detect statistically significant differences across consortia.
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
481
THE JOINT COMMISSION Team Participation All 38 hospitals in the final sample—those that had used consortium services for CQI training and had completed a pilot project—had formed crossdisciplinary teams for the project they were describing. Teams ranged in size from 3 members to 23; at least two disciplines were represented on each team. Mean team size was 7.4 (standard deviation [SD], 3.99), with a median of 6.5. Team composition by discipline was highly variable (the quality managers made the disciplinary classifications, choosing from the categories presented). Quality managers participated in 44.7% of the teams, laboratory staff in 29.0%, administrators in 55.3%, physicians in 60.5%, and nurses in 65.8%. Staff in the “other” category—including personnel such as radiology technicians, dietitians, respiratory therapists, chaplains, and volunteer coordinators—participated in 89.5%. Employee empowerment is one of the five elements of CQI processes that we attempted to measure. We assessed this dimension using a number of items that were borrowed from previous work assessing degree of implementation of CQI in hospitals.9 Team members identified the issue their team addressed and selected the project 64% of the time, the methods of analysis 87% of the time, and the approaches to resolving the problem or issue 97% of the time. Twenty-seven (75%) of the 38 respondents agreed with the statement that team members had the authority to resolve the problem without appealing to higher levels of management. This appears to be a relatively high level of employee empowerment and involvement in decision making perceived by quality managers. Types of Projects CQI projects, listed in Table 4 (p 483), were classified on the basis of the quality manager’s judgment as to whether they were primarily clinical or nonclinical in nature. We also asked quality managers to rate each project on a five-point scale in terms of how critical the outcomes of the project had been, first for clinical services in the hospital, and second for nonclinical services. In only 3 of the 38 projects did the quality managers consider the problems to be unresolved (and the projects therefore ineffective)—too small an n, we
482
believed, to enable us to explore differences between effective and ineffective projects. In terms of the ratings of the importance of the projects’ outcomes, we collapsed each question into a dichotomous outcome (not critical/critical) and then combined the two into a single variable, in which we counted the outcome of the project as critical to the hospital if it was deemed critical either to clinical or to nonclinical services. Quality managers described 69% of project outcomes as critical to clinical services. The projects varied considerably in topic. Some of the projects (for example, one focusing on a transitional care program) dealt with a specific program within a hospital. Others dealt with hospitalwide issues, such as paperwork and documentation requirements across the entire facility. A few focused on a specific health problem or service, such as pressure ulcers or epidural anesthesia. Most of the projects had some direct relationship to patient care, although a few (such as the redesign of an award and employee recognition project) focused on issues that have little or no direct effect on patient care. We examined bivariate correlations between several hospital characteristics and whether a project was considered primarily clinical by the quality manager. We found that a hospital being private is directly correlated with the project being considered primarily clinical; the status of CQI implementation three years previously, use of CQI prior to training by the consortium, and amount of progress on implementing CQI processes were not significantly correlated with whether the project was considered primarily clinical. Costs and Duration of Projects Costs for the CQI projects varied widely, from $148 for the entire project to $18,590 (mean, $5,047 [SD, $4,680]; median, $3,580). The length or duration of the projects also varied widely, from 1 month to 66 months (mean, 10.7 months [SD, 11.9]; median, 8 months). It was difficult to compare costs for projects without using project duration as a denominator. We also calculated a measure of the labor intensity of the project, defined as the total personnel hours divided by the number of months the project lasted (person hours/month). The mean and median total costs, duration, cost per month, and
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Table 4. Projects Included in the Study (N = 38)*
Project considered by quality manager to have critical outcome for either clinical or nonclinical services
Project title Project is considered primarily clinical Early discharge program for normal vaginal delivery Paper chase Pathology report turnaround time Identification and treatment of patients with chest pain in emergency room Patient assessment Chest pain protocol Epidural anesthesia Point-of-care glucose testing Document turnaround time Efficiency of medical staff sections Same-day surgery process Patient transport team Food delivery project Weight team Pressure ulcers Home health discharge planning Ventilator management Unavailability of diagnostic imaging films Timely reporting of panic lab values Conscious sedation
Yes Yes Yes Yes No Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No Yes
Project is considered primarily nonclinical Managing emergency room transfers to other facilities Medical records task force Redesign award and employee recognition Transitional care program Ineffective billing process Interagency patient transfer Chat book Timeliness of ancillary tests in emergency room Paper flow pit Patient satisfaction Receiving and shipping Outpatient clinic wait time Lab consolidation Engineering work order team Blood wastage Antibiotics on time Reduction in time to administration of thrombolytic therapy in emergency room Operating room turnaround time
Yes Yes Yes Yes Yes Yes No No Yes Yes Yes Yes Yes No No No No No
* Project titles in italics indicate problems considered unresolved by the quality manager.
VOLUME
26
NUMBER
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
483
THE JOINT COMMISSION Table 5. Costs, Duration, Costs per Month, and Labor Intensity of CQI Projects, by Consortium*
Consortium Vermont: Mean (SE) Median Iowa: Mean (SE) Median Utah: Mean (SE) Median California: Mean (SE) Median
Total cost
Duration in months
Cost per month
Labor intensity (total FTEE person hours/month)
$4,430 (1,139) $2,700 $4,902 (1,589) $3,980 $2,898 (746) $3,580 $6,562 (2,125) $5,190
8.0 (1.2) 7.0 19.1 (7.3) 15.5 7.2 (3.1) 3.0 8.8 (1.7) 10.0
$931 (311) $465 $543 (285) $261 $809 (413) $210 $1,112 (488) $969
33.9 (13.7) 10.7 24.0 (10.4) 14.3 36.2 (15.2) 13.5 32.1 (11.6) 25.0
* FTEE, full-time equivalent employee; SE, standard error of the mean.
intensity of the projects in each consortium are shown in Table 5 (above). The CQI projects varied considerably across sites in total costs, monthly cost, and labor intensity, although none of these differences was statistically significant. In addition to exploring the differences in cost, duration, and intensity of the projects by consortium, we examined differences in these elements between projects that quality managers considered to have critical outcomes either for clinical or nonclinical services in the hospitals. While there are very large differences between the means for these two groups of projects, none of the differences were significant, again because of the very wide variation in the distributions of the variables in both groups (Table 6, p 486). As in the previous discussion, the sample size in this study is too small to detect statistically significant differences. However, it appears that the observed higher costs in the group of projects deemed to have critical outcomes might be related to the higher intensity (person hours/month), as well as longer duration of the projects.
Discussion We have presented a detailed picture of the costs and use of CQI processes among a set of CQI projects, sampled from all the projects completed and under way in 38 hospitals involved in the IQHC program. Lack of adequate power because of small sample size was an important limitation of this study.
484
Sample size considerations might limit the likelihood of further research in this area, but it is important to consider the issue. Quantitative analysis in CQI is rife with small-sample problems and low underlying rates. For example, low-denominator numbers (such as the number of diabetics in a given practice or hospital to whom a specific intervention applies, or as in mortality rates, which makes it difficult to use mortality as an outcome in a small-area study) make it important to think carefully about appropriate tests of statistical significance, as Diehr and colleagues have pointed out with respect to small-area analysis.14–16 Given the nature of CQI processes and their focus on individual projects, it is difficult to accumulate a sufficient number of observations to enable statistical inference.* There are some interesting lessons to learn from the data collected for this study. The first is that in the hospitals included in this sample, all of which were highly self-selected (evidenced by their participation * Because this is the first study to report costs of team processes for individual CQI projects, it is instructive to calculate the sample sizes needed to achieve a probability of statistical significance in the difference between mean costs for CQI projects in one setting compared to another. The difference that will be used in this example is the difference between the mean cost of CQI projects in Vermont compared to those in Iowa: Mean costs in Vermont are $4,430 (SD, $1,139), and mean costs in Iowa are $4,902 (SD, $1,589). The difference in the means is $472. With power set at .90 and alpha at .05 for a two-sided test of statistical significance, the minimum number of CQI projects in each sample required to detect the observed mean difference would be 181. We included 31 projects in our sample, each from a different hospital. If projects from the same hospital were included, it might be necessary to correct estimates for cluster sampling, requiring an even larger sample size.
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
in a voluntary consortium of hospitals focused on quality of care), knowledge of CQI processes appears to be fairly thorough, at least from the perspective of quality managers.* Adherence to five basic tenets of CQI, described at the beginning of this article, is evident in most of the projects described. Many of the hospitals, by the time we conducted this survey, were fairly advanced in their implementation of projects based on CQI. The use of cross-disciplinary teams appears to be fostering collaboration among different disciplines in hospitals, one of the primary goals of CQI. Teams appear to have a reasonable amount of autonomy, at least in the view of quality managers in the hospitals. It is interesting to note that some of the core disciplines (nurses and physicians) providing health care in hospitals did not appear to be represented in large numbers on CQI teams (at 18% and 24% of total team membership, respectively), whereas administrators and other staff were relatively well represented (15% and 35%, respectively). Physician participation in CQI projects is often considered difficult to achieve,17 yet when physicians were employees (in this study, primarily in the California consortium), rates of physician participation were higher than when they were predominantly either self-employed or employed by an organization other than the hospital. Participation by hospital employees was generally considered part of the usual work schedule. Physicians tended to spend the least amount of time in project work, while other types of staff generally participated equal amounts of time. This probably represents the attempts of hospital staff to minimize the time costs for physicians of CQI team work and is probably one of the balancing acts necessary to gain their participation in an activity that is valued by the organization. In terms of costs of projects, however, limiting physician involvement and time commitment is a way of keeping project and team costs lower than they would be if physicians spent a great deal of time on the
projects. Physician hourly rates were among the highest wage rates of any discipline involved in projects.† Administrators were next highest in reported wage rates, and, like physicians, tended to spend less time on projects than other team members. Although the average costs of the team work needed for CQI project completion do not seem extremely high, it is important to remember that most hospitals conduct several CQI projects a year, and especially in large hospitals, have multiple CQI projects going on at one time. If the average costs reported in this article were multiplied by five to ten projects a year for a hospital, the costs of team work needed to conduct CQI projects begins to appear higher. In addition, it is worth bearing in mind that there is usually a great deal of additional work not related to CQI projects being carried out by quality management staff, which may include monitoring various indicators of quality of care such as nosocomial infection rates, and conducting required assessments and reviews of systems required for Joint Commission accreditation. Managing quality of care is a complicated business for most moderate and large hospitals and health care organizations, even without CQI processes and their demands. Adding costs of CQI teams to “routine” quality management or quality assurance may, for hospitals facing severe budget pressures, prompt examination of the value of quality management activities as a whole. In the mid-1990s, at least in some public hospitals in California, the results of such an examination were to eliminate the quality management staff. Although CQI processes have only relatively recently been adopted by health care organizations, they have come to be regarded as staples of quality management in most hospitals in the United States. This is in part because of mandates and perceived mandates by accrediting bodies, and in part because many hospital employees who receive training in the processes and techniques of CQI become very enthusiastic about its potential to help in dealing with problems in quality of care.
* We did not examine the extent to which staff throughout the hospital had received training or participated in CQI activities. In previous research, which included some of the same hospitals as this sample, we found that in about 30% of hospitals surveyed, more than 50% of the staff had received some type of CQI training.
† We collected information about physician hourly rates whether the hospital paid for the physician’s time or not, since we were interested in assessing as much of the cost of CQI projects as possible. Costs for the projects were mostly driven by hourly rates and time spent on team meetings.
VOLUME
26
NUMBER
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
485
THE JOINT COMMISSION Table 6. Costs, Duration, Costs per Month, and Labor Intensity of CQI Projects by Whether the Project Outcome Was Considered Critical*
Program outcome considered critical? Projects considered critical: Mean (SE) Median Projects not considered critical: Mean (SE) Median
Total cost
Duration in months
Cost per month
Labor intensity (total FTEE person hours/month)
$5,114 (4,413) $4,166
11.18 (12.72) 10.76
$868 (1,069) $294
17.43 (13.1) 11.9
$4,854 (5,671) $2,675
8.83 (2.99) 9.0
$494 (424) $234
24.0 (10.4) 14.3
* CQI, continuous quality improvement; FTEE, full-time equivalent employee; SE, standard error of the mean.
However, the adoption of CQI in hospitals has received relatively little systematic study to date. This is due, in part, to the difficulty of studying these processes in ways that permit comparative quantitative analysis. Our findings indicate that a large number of projects would need to be included in a comparative study to fully examine the factors that influence the costs of CQI projects. In addition, further research on whether CQI projects ultimately affect patient outcomes, and whether they are cost-effective compared to other methods of managing quality of care, is required. Several approaches to designing studies of CQI cost-effectiveness could be developed. One might be to design a study that collects data on several CQI projects for each hospital in a representative sample of hospitals that have been completed within, for example, the past two years. The number of CQI projects that we estimate would be needed to detect statistically significant differences in costs of projects, based on our findings, is about 180. Although it would not be difficult to accumulate this number of projects in a hospital sample similar to the one in this study (38–40 hospitals), it would be necessary to perform cluster correction to ensure that statistical inferences appropriately adjust for including more than one study per hospital in the sample. Using outcomes from a cluster sample such as this to evaluate consortia, like the ones in our study, would probably require more than the 180 or so projects we suggest would be necessary if the sample of projects came from different hospitals. CQI is one model for managing and improving quality of care in health care organizations. Alternative and sometimes complementary models include performance management, in which the focus is on outcomes
486
rather than process, and more “traditional” methods of quality assurance, including a focus on indicators (such as nosocomial infection rates) of quality of care. While none of these models provide a final solution to how quality of care can be managed and ultimately improved, we should regard these as technologies to be assessed by the same criteria (such as cost-effectiveness) used to assess other technologies. In an era of cost containment in health care, questions related to the costs of doing business must be addressed. While our study cannot provide definitive answers to questions about how effective or cost-effective CQI processes are in hospitals, some of our findings suggest approaches that might be useful to quality managers. First, new CQI projects should be subjected to scrutiny in terms of their likely contribution to quality of care, as distinct from other positive outcomes (for example, improving efficiency in nonclinical services, enhancing the public image of the hospital or health care facility). If the primary goal is improved direct quality of care, then the project should have goals and objectives that foster that as an outcome. The composition of the team should also be carefully assessed. Because of heavy work loads and demands on time by all staff in a hospital, it is often tempting to include people who might not have direct knowledge of the processes to be improved as part of a CQI team, to provide additional human resources. While this is sometimes helpful, it should be recognized that it may extend the time required to complete the project, and the additional costs associated with longer duration of a project should be taken into account as teams are formed and their objectives are determined.
AUGUST 2000 JOURNAL Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
JOURNAL
ON
QUALITY IMPROVEMENT
Commitment from team members is a critical element in timely completion of a project, and if key participants are not able to make a commitment to a project, there should be careful review of whether the project should be chartered. CQI tools and techniques have largely been embraced in health care, despite the fact that to date, relatively little comparative empirical research exists on the effectiveness of these techniques.18 More recent studies that have been published8,12 find mixed results, indicating that it is difficult to demonstrate that CQI as a dominant method of managing QI is
both effective and cost-effective. However, since few of the other methods of quality management have received the same scrutiny, it is difficult to make the necessary judgments as to how best to improve or manage the quality of care in health care settings. During the past decade, we have improved our capacity to measure quality of care and its outcomes. We need to apply this improved capacity to examining the technologies we use to manage quality of care, measuring the effectiveness of different approaches, and exploring the relative costs of different ways of managing quality. J
References 1. Berwick D: Continuous improvement as an ideal in health care. N Engl J Med 320:53–56, 1989. 2. Laffel G, Blumenthal D: The case for using industrial quality management science in health care organizations. JAMA 262:2869– 2873, 1989. 3. O’Leary DS: Accreditation in the quality improvement mold: A vision for tomorrow. Qual Rev Bull 17:72–77, 1991. 4. Kabcenell AI, Cohen AB, Merrill JC: Importing a model of hospital quality from the Netherlands. Health Aff 10:240–245, 1991. 5. Sales A, et al: Is quality in the eye of the beholder? Jt Comm J Qual Improv 21:219–225, 1995. 6. Sales A, Moscovice I, Lurie N:
VOLUME
26
NUMBER
Measuring seriousness of hospital quality of care issues. Jt Comm J Qual Improv 22:811–816, 1996. 7. Dranove D, et al: The cost of efforts to improve quality. Med Care 37:1084–1087, 1999. 8. Shortell SM, et al: Assessing the evidence on CQI: Is the glass half empty or half full? Hosp Health Serv Admin 40(1):4–24, 1995. 9. Shortell SM, et al: Assessing the impact of continuous quality improvement/total quality management: Concept versus implementation. Health Serv Res 30:377–401, 1995. 10. Lammers JC, et al: Total quality management in hospitals: The contributions of commitment, quality councils, teams, budgets, and training to perceived improve-
ment at Veterans Health Administration hospitals. Med Care 34:463– 478, 1996. 11. Ziegenfuss JT, et al: Engineering quality through organization change: A study of patient care initiatives by teams. Am J Med Qual 13(1):44–51, 1998. 12. Shortell SM, et al: Assessing the impact of total quality management and organizational culture on multiple outcomes of care for coronary artery bypass graft surgery patients. Med Care 38:207–217, 2000. 13. Traska MR: Don’t try total quality improvement if you think it will solve problems it shouldn’t. Healthcare Competition Week 9(6): 9–10, 1992. 14. Diehr P, Grembowski D: A
small area simulation approach to determining excess variation in dental procedure rates. Am J Public Health 80:1343–1348, 1990. 15. Cain KC, Diehr P: Testing the null hypothesis in small area analysis. Health Serv Res 27:267– 294, 1992. 16. Diehr P, et al: 1992. Can small-area analysis detect variation in surgery rates? The power of small-area variation analysis. Med Care 30:484–502, 1992. 17. Shortell SM, et al: Remaking Health Care in America: Building Organized Delivery Systems. San Francisco: Jossey-Bass, 1996. 18. Bigelow B, Arndt M: Total quality management: Field of dreams? Health Care Manage Rev 20(4):15–25, 1995.
8
Copyright 2000 Joint Commission on Accreditation of Healthcare Organizations
487