- Email: [email protected]
An Initial Assessment of the Veterans Affairs Occurrence Screening Program
f
,
Ronald L. Goldrnan, PhD, MPH Debby J. Walder, RN, MSN
An Initial Assessment of the Veterans Affairs Occurrence Screening Program
Identification o f poor-quality care remains a critical issue for hospitals and other health care organizations. Many organizations have continued to rely on certuin screening criteria thut are associated with preventable adverse events. Yet there remains little research on the sensitivity and specificity of such criteria. No one knows what the optimal set of screens is. This article by Goldman and Walder represents the kind of research that is leading us to more rational selection of screens. Relying on feedback from hospitals as well as a number of other independent analyses, the authors provide a good deal of insight into the utility of various screens. Their discussion of the costs of screening and the reliability of screeningjudgments is extremely useful. Although much more research is needed, Goldman and Walder appear to be moving us in the right direction. noyen A. Brennan, MD, JD, MPH, Professor of Law and Public Health, Harvard School of Public Health, and Associate Professor of Medicine, Harvard Medical School, Boston.
ccurrence screening has been defined a s the professional review of cases that involve adverse outcomes to identify opportunities for the improvement of care.' The Department of Veterans Affairs (VA) established an occurrence scmning program in its 159medical centers in October 1988;the program was designed to be consistent with the provisions of Public Law 100-322, which mandated the implementation of occurrence screening throughout the VA. This article will present data on the operation and effectiveness of the program and will describe recent modifications made in the program in response to these data.
Ronald L. Goldman, PhD, MPH, is Program Evaluator, Department of Veterans Affairs Central Office, Washington, DC. Debby J. Walder, RN, MSN, is Clinical Program Manager. Please address requests for reprints to Dr Goldrnan at the Office of Quality Management (15),Department of Veterans Affairs, 810 Vermont Avenue, NW, Washington, DC 20420. The authors wish to thank Thy Churchman for his work on data input and editing and three anonymous reviewers for their helpful comments.
Description of Original Program
Each medical center was required to apply the same nine screening criteria (Figure 1, p 328)to all acute care inpatients and to include the following four general stages in its program: 1. Screening to identify cases that meet the criteria; 2. Clinical review t o determine whether any aspects of these cases require further review; 3. Peer review to determine if practitioner care was less than optimal or, alternatively, review of system or nonpractitioner issues by an appropriate service or committee; and 4. Administrative review to determine the appropriate corrective action when peer review indicated that practitioner care was less than optimal. Within this general framework, medical centers selected the specific procedures so that the program could be adapted to local circumstances. VA policy developers attempted to minimize the resources required for the program by choosing screening criteria defined in terms of information avail-
able within the VA computer system. This allowed for automated identification of most cases requiring raiew, eliminating the need for manual review of medical records to identify occurrences, perhaps the most labor-intensiveaspect of other occurrence screeningprograms. Most exceptions to the criteria could not be automated and are identified through manual review.. To further reduce resource requirements, the developers established the clinical review function to minimize the amount of physician time needed to review cases. Data Collection
Monitoring of the implementation and effectiveness of the occurrence screening program has been continuous since its inception. The primary data collection instrument is a semiannual survey form that the quality management office at each'hospital completes after gaining input from other facility personnel and mails to the central office. Each report includes data on the results of the review process for each screening criterion, that is, the number of occurrences, the number of occurrences sent to peer review, and the peer review findings. The data for the period October 1989-March 1990were edited most intensively since staff were available at that time. This survey also included the most detailed questions about the implementationand effectiveness of the overall program and about the usefulness of each of the individual criteria. For these two reasons, data for that sixmonth period are presented here. The facility reports for that period were edited to identify the responses that were significantly different from those of comparably sized hospitals or from the data that the medical center had submitted for other six-monthperi-
Occurrence Screening Criteria Element 1. Readmission within 14 days'
2. Admission within 3 days following unscheduled ambulatory care visitt
3. Admission within 3 days following ambulatory surgery procedure* 4. Admission or absent sick in hospital from Veterans Affairs nursing home care unit* a. within 14 days of discharge from acute care b. to psychiatry service 5. Transfer from intermediate medicine* a. within 14 days of transfer from acute care b. to psychiatry service 6. Transfer to a special care units a. within 72 hours of transfer from a special care unit b. within 72 hours of a surgical procedure
7. Return to operating room in same admission11
8. Cardiac or respiratory arrest 9. Death#
Exceptions Readmission was scheduled at time of discharge Prior discharge against medical advice or irregular Scheduled admission Admission same day as visit Scheduled admission None
None
Planned transfer after surgery Transfer after emergency operation, which ordinarily requires transfer to special care unit Special care unit used as recovery room after surgery Planned transfer for procedure such as cardioversion Transfer from one special care unit to another Planned return to operating room documented prior to first surgery Two operations separated by more than 60 days Did nbt survive arrest Do-not-resuscitate (DNR) order or local equivalent at time of death Admitted for palliative care
'In a recent program revision criterion 1 was changed to readmission within 10 days. Two new exceptions are (1) readmission for alcohol or drug abuse, chemotherapy or radiation therapy; and (2) condition precipitating readmission did not exist at time of prior admission. t A new exception to criterion 2 was added: admission to Psychiatry. $Criteria 3, 4, and 5 were deleted. §Criterion 6 was modified to include only transfers to a special care unit within 72 hours of transfer from a special care unit. The exceptions are (1) planned transfer, and (2) transfer from one special care unit to another. IIA new exception to criterion 7 was added: second procedure unrelatedto first. #The first exception to criterion 9 was changed to DNR order at time of admission or more than 7 days prior to death.
Figure 1.Each medical center in the study was required to apply these nine occurrence screening criteria to all acute-care inpatients.
ods. We contacted the hospitals and asked to check their responses. A number of major errors were identified in this manner; these primarily concerned the number of occurrences identified. For 11hospitals it was not possible to correct inaccurate data retrospectively; we deleted these facilities from analyses involving the results of the review process (Figure 2, p 329). Two other hospitals had not sufficiently implemented
the program by October 1989to provide usable data for these analyses. Results Figure 2 shows the results of the review
process for the remaining 146facilities. Eighteen percent of the occurrences identified during the reporting period were sent to peer review. 'henty percent of the peer-reviewed occurrences received Level 2 or 3 ratings indicating
practitioner-related quality concerns. (See the practitioner quality-of-care scale in n b l e 1[p 3301 for definitions of Level 2 and 3 peer review ratings.)About 350 confirmed system/equipment issues were found. In all, opportunities for improvement related to practitioners or system/equipment issues were found in 4.2% of all occurrences.* There was considerable variability among medical centers. The proportion of occurrences referred to peer review was 8.6% for the 25th percentile and 212% for the 75th percentile. For the proportion of occmnces in which quality concerns were identified, the 25th and 75th percentiles were 1.7% and 7.9%, respectively We used a widely employed typology developed by Stefos et al to examine whether the results of the review process varied across types of medical centers. These authors employed cluster analysis to assign VA medical centers to one of six groups based on work load, scope of services, medical complexity of patients treated, and labor costs. The six types of hospitals are Group 1 (Small Affiliated), Group 2 (SmallGeneral),Group 3 (MidsizeAffiliated), Group 4 (Midsize General), Group 5 (LargeAffiliated),and Group 6 ( Psychiatri~).~ n b l e 2 ( p3:l) presents the findings of the analyses using these hospital groups. The overall variation among the six groups for both the proportion of occurrences in which opportunities for improvement were identified and the proportion referred to peer review was highly statistically significant (P = 0.0000),in part because of the very large sample sizes. As can be seen in %bled 2, the variation in identifying opportUnities for improvement was due to more frequent identification of such opportunities among Group 1 (Small Affiliated), Group 2 (Small General), and Group 6 (Psychiatric),the three groups with relatively small medical and surgical services and relatively few occurrences. The combined proportion for these three groups is 6.05% as com-
1
*The number O f occurrences at each facility was corrected fir theproportion of occurrences referred to peer review for which peer had not been completed when the report was submitted.
1
,
pared to a rate of 3.58% for the other three groups (X2 = 152.44, df = 1, P = 0.0000). The facilities in Groups 1, 2, and 6 also referred a larger proportion of occurrences for peer review (X2= 270.11, df = 1, P = 0.0000). However, this effect is largely due to the high rate shown by Group 1. To explore the perceived ektiveness of the program, we asked each facility how effective the occurrence screening program had been at identifying qualityof-care issues compared to other elements of their quality assurance (now termed quality management)program. The percentage of facilities that chose each of the five possible responses is listed in n b l e 3 ( p 331). 'Ib summarize briefly, 56% of the facilities selected "Most Effective Element" or "One of More Effective Elements," 24% chose "About Average," and 20% chose a response indicating the program had less-than-average effectiveness. The overall variation among the six hospital groups in the perceived effectiveness of occurrence screening was not statisticallysignificant at the 0.05 level; in particular, there was no tendency for Groups 1,2, and 6, the groups containing facilities with smaller medical and surgical programs, to evaluate the program more positively. Opportunities to improve care were attributed to the occurrence screening program by almost all of the facilities in response to a question on the survey asking for examples of such opportunities. Nearly one-half of the respondents described changes in policies and procedures that had resulted from the analysis of occurrence screening data. For example, several facilitiesdescribed changes in procedure intended to reduce the incidence of aspiration pneumonia among patients receiving enteral feedings.About one-third of the respondents described either interventions into the care of patients being reviewed or the development of improved methods for tracking the care of future patients, for example, procedures for the follow-up of suspiciouschest x-raysor critical laboratory findings. Another third of the facilities reported changes in medical record documentation. Staff education, especially pertaining to drug utilization, drug interactions,and drug toxicitx was
Results of Occurrence Screening Review Process for 146 Veterans Affairs Hospitals from October 1989 through March 1990 435,000 acute-care inpatient episodes
1 57,841 occurrences
10,698 referred to peer review ( 1 8.5%)
1 359 confirmed system1
9,412 completed peer reviews (88.0%)'
1 'The results of some peer reviews were not available when the reports were submitted by the hospitals.
Figure 2. This chart breaks down the results of the occurrence screening review process.
also mentioned by approximately onethird of the respondents. Enhanced staffing levels, the acquisition of new equipment, expansion of support servips, establishment of specialty arnbulatory care clinics, and development of clinical protocols were other examples frequently reported. Several facilities mentioned that additional equipment and staffing had been specifically dedicated to the monitoring of patients receiving sedation during procedures. Respondents also mentioned a number of times changes in the management of intravenous sites and lines to reduce the incidence of infections. Medical center responses also emphasized several general themes. A number of hospitals indicated that the program had increased the interest and involvement of physicians in peer review and that this would have a long-term positive impact on a wide range of quality management activities. Improvements in resident supervision and staff communication were also often attributed to the occurrence screening program. Effectiveness of Individual Screening Criteria Each facility was asked to rank the criteria with regard to usefulness in iden-
tifying opportunities to improve care. Since 3 of the 9 criteria involved two parts, there were, in effect, 12 criteria to rank. However, some criteria were not applicable to every hospital-for example, the criterion concerning returns to the operating room could not be used a t fa~ilities where surgery was not performed. n b l e 4 ( p 332) presents the proportion of facilities ranking each criterion among the three most useful. Readmissions ( # I ) ,death (#9),and admissions within three days of an unscheduled outpatient visit (#2),the three most effective criteria, were each ranked highly by 50%or more of the facilities. Return to the operating room ( #7) and cardiac or respiratory arrest (#8)were ranked among the top three criteria by a smaller, but still substantial, number of hospitals. Most of the remaining seven criteria were ranked as one of the three most useful criteria by no more than a few facilities. The large differences in the percentages of hospitals that ranked different criteria strongly indicate considerable agreement among the facilities with regard to the effectivenessof the 12 criteria. Not surprisingly,almost no differences were found between the ratings made by the six hospital groups
Table 1. Practitioner Quality-of-Care Scale
previously described; the major exception is that facilities in Group 6 (Psychiatric) gave relatively low ranks to return to the operating room ( #7) and return to a special care unit (#6a),criteria that have little relevance to them. A limitation of this analysis is that the appropriate interpretation of a criterion's rank as one of the three most effective by a hospital depends upon the total number of criteria used by that facility To control for this artifact, the rank each criterion received from a hospital was divided by the number of criteria used by that facility. We then computed the means of these adjusted ranks for each criterion. The results, which are shown in the last column of lkble 3, are almost identical to those found in the original analysis. Staffing Requirements of Program
The primary source of data on the staffing requirements of the occurrence screening program is a staffing survey designed to identify personnel whose primary responsibilitywas quality management. The VA's Office of Quality Management conducted this survey from April 1989 to July 1989, shortly after the inception of the occurrence screening program. One purpose of this survey was to ascertain the number of full-timeemployee equivalents( FTEEs ) a t each facility who were directly involved in specific quality management activities such as occurrence screening. For purposes of the survey, occurrence screening was defined in terms of the mandated screening criteria as well as any additional criteria the hospital had
have been a useful component of the VXs overall quality management effort at the time of data collection. The program, which was implemented throughout the might have handled system, resulted in the identification of large numbers of opportunities for improvement. Although not every facility had a good experience with occurrence screening, a majority of facilities regarded the program as one of the more effective elements in their overall quality management programs. 2. The staffing costs of the VA pro1 gram appear to be less than those 1 reported for occurrence screening pro- I grams in which manual screening is p e r f ~ r m e d . This ~ . ~ increased costeffectivenessis probably a consequence chosen to use. Only staff whose total of the use of computers for case identiinvolvement in quality management fication. Although it was initially diffiactivities represented 0.5 FTEE or more cult to develop software that met the were included in the survey Thus, the needs and expectations of all facilities, time of physicians and other staff who the use of computerized case identifireviewed records for the occurrence cation and data analysis has been a screening program, but who primarily highly positive aspect of the program. 7 performed clinical duties, was not These conclusionshave to be balanced j included. The median number of staff who worked in the occurrence screen- against several known important lirnitations of occurrence screening. Even ing program at the 128 hospitals with with the use of computerized screenfully implemented programs at the time of the staffing survey and that had com- ing, it is a resource-intensiveprogram plete$ the relevant portions of the staff- in which quality issues are identified ing survey was 0.7 FTEE. Seventy-two in only a small proportion of cases reviewed-4.2% in this report. In addipercent of these 128 facilities used 1.0 tion, peer reviey is central to occurrence or fewer staff in their program; 17% between 1.01 and 2.0 staff; and 11% screening; the available evidence indi- , cates that peer review, as usually pracmore than 2.0 staff. ticed, has quite limited interrater We also examined the relationship r e l i a b i l i t ~Of ~ . particular ~ relevance is I of the number of staff in a facility's prothe recent data reported by Rubin et a1 i gram to other variables. The number questioning the accuracy of peer reviews of occurrences, which indicates the work conducted by peer review organization load associated with the program, showed the strongest association (Spear- physician reviewers within the Health Care Financing Administration (HCFA) man rank correlation = 0.41, P = .~ 0.0001).There was also a significant rela- generic screening p r ~ g r a r n Third, ocamence screening programs typically tionship with the number of cases sent to peer review, another variable reflect- focus on deficient practitioner care,making work load (Spearman rank correla- ing it difficult to gain the enthusiastic involvement of clinical staff. FinaU3: the tion = 0.19, P = 0.04). There was no available data do not unequivocally supassociation with the measures unrelated port the effectiveness of screening crito work load, such as the percentage of teria in identifying cases for review that occurrences referred to peer review or are particularly likely to have quality the proportion in which opportunities concerns.69The recent finding of Rubin for improvement were identified. et al in which the HCFA generic screens were only slightly better than random Discussion sampling at targeting appropriate cases Several conclusions are suggested by for physician review is particularly notathe findings: ble in this regard.6 1. Occurrence screening appears to
'
1[ k
i P
$ i
t
The Office of Management has recently made a number of changes in the occurrence screening program to address the first three concerns. First, criteria 3 , 4 , 5 , and 6b have been dropped because of their relatively limited effectiveness;several of the remaining criteria have been refined to increase their yield. For example, the time period for the readmission criterion ( #l) has been changed from 14 days to 10 days; this modification is based on the recent finding of Ranki et a1 that almost all preventable readmissions occur within the shorter period.1° Similarly, the primary exception for the death criterion ( #9) has been changed from all do-notresuscitate (DNR)cases to only those in which the DNR order existed at the time of admission or more than 7 days prior to death. This change was based primarily on Dubois and Brook's finding that patients who had DNR orders written after admission often had preventable deaths." A second change is that annual evaluations of the interrater reliability of clinical and peer reviews are now being performedat each hospital. If agreement between reviewers is below a specified level, action to improve the reliability of the assessments is required. Finally, to expand the range of the program, the review process is now being used to identify exemplars:as well as deficient, practitioner care; greater emphasis is also being placed on the identification of opportunitiesfor improving care that involve system issues as recommended by Brooks et a1.12We are hopeful that these changes will enable the congressionally mandated VA program to overcome some of the usual shortcomings of the occurrence screening process. There are several limitations of this study. First, the data are from 1989and 1990; it is unclear to what extent they can be generalized to the present. However, analyses of more recent but less intensively edited data concerning the results of the review process (Figure 2) produced results highly similar to those reported here. Second, the data were self-reportedby the participating hospitals, and we could only edit them for major inaccuracies. After gaining input from other facility personnel, the survey forms were for the most part com-
Table 2. Results of Review Process for Six Groups of Veterans Affairs (VA) Medical Centers
Table 3. Responsesto Questions about Effectiveness of Occurrence Screening at Identifying Quality-of-Care Issues
pleted by quality management staff because they were the most knowledgeable sources of information. Their close involvement with the occurrence screening and other quality management programs may have biased their responses.* Finally, guidelines were not provided to the facilities to use in ranking the criteria regarding their usefulness in identifying opportunities for improvement nor were the facilities asked to explain their rankings. Thus, the data do not enable us to determine why some criteria were considered to be more *However, brief narmtive assessments o f the program by chiefs of stafi which were included in the same survey, were consistent with the ratings of quality management staff.
effective than others in generating quality improvements. Some commentators have suggested that occurrence screening should be largely replaced by methodologiesthat only involve peer assessment when a provider exceeds predetermined threshold, shows a negative trend, or is a statistical outlier in a comparativeanalysis.*." Since these approachesgenerally require less professional time, they may be more cost-effective. However, recent evidence suggests that occurrence screening procedures can be quite sensitive in identifying cases involving care that is significantlyproblematic. For example, Brennan et a1 showed that approximately 80% of malpractice claims involving medical causation of adverse
Table 4. Rankin
cations described in this paper. These studies should enable the VA to conclusively assess its occurrence screening program.
riieria
References 1. Goldman RL: Development of a Veterans Administration occurrence screening program. QRB 15:315-319, 1989. 2. Stefos T, LaVdee N, Holden F: Fairness in prospective payment: A clustering approach. Health Sen, Res 27:239-261,1992. 3. Craddick J W Improving Quality and Resource Management Through Medical Management Analysis. Rockville, MD: Medical Management Analysis International, Inc, 1987. 4. Bittle and Associates: Criteria Based Review for Integrating QA/RM/UR Programs. Baltimore, 1987. 5. Goldman RL: The reliability of peer assessments of quality of care. JAMA 266:958-960, 1992.
events or negligence could be identified through an occurrence screening review of the medical r e c ~ r dThus, .~ a well-run occurrence screening program should enable a hospital to identify many of its opportunities for improving patient care. The long-term role of occurrence screening and other methods involving review of individual cases will be in part determined by studies that show what proportion of these opportunities would also be uncovered through more statistically oriented approaches. In the only articlerelevant to this issue of which we are aware, McGuire et al reported that statistical analyses of surgical outcomes were less efktive in identifying patient care problems and in improving care than was individual case review for a 14-year period a t one facility.15 A second consideration is that the regular review of large numbers of individual cases in occurrence screening programs may indirectly improvequality simply because practitioners are aware that their care in any given case is likely to be reviewed. Thus, the costeffectiveness of occurrence screening may be quite different that the 4.2%
yield presented above suggests. The magnitude of the "surveillance effect" produced by different quality assessment methods is another important question for future research. Any definitive conclusion concerning the usefulness of occurrence screening can only be determined by studies that compare its cost-effectiveness to that of other quality management programs. Accordingly,we are examining the improvements in care that have resulted from occurrence screening and other VA quality management programs at a representative sample of facilities. The number and importance of the improvements in care resulting from each program will be related to the program's costs so that the costeffectiveness of the different programs can be compared. A second evaluation project will assess the sensitivity and specificity of peer review within the occurrence screening program. Next year, non-VA physician panels from the VAh new External Peer Review Program willindependentlyexamine 1,700 cases that had been previously peer reviewed. The VA is also planning to evaluate the impact of the recent program modifi-
6. Rubin HR, et al: Watching the doctorwatchers: How well do peer review organization methods detect hospital care quality problems? JAMA 267:2349-2354, 1992. 7. Barnes C, Moynihan C: Accuracy of generic screens in identifying quality problems: Analysis of false-positive and false-negativeo c c ~ c e slbpics . in Health Records Management 9:72-80,1988. 8. Brennan TA, et al: Identification of adverse events occurring during hospitalization. Ann In* Med ll2:221-226,1990. 9. Hannan EL, et al: A methodology for targeting hospital cases for quality of care record reviews. A m J Public Health 79: 430-436, 1989. 10. Ranki SE, Breeling JL, Goldman L: Preventability of emergent hospital readmission. A m J Med 90:667-674, 1991. 11. Dubois RW, Brook RH: Preventable deaths: Who, how often and why? An? Intern Med 109:582-589, 1988.
12. Brooks JHJ, et al: Systems versus performance problems: A peer review organization's perspective. QRB 18:172-177,1992. 13. Sanazaro PJ, Mills DH: A critique of the use of genericscreeningin quality assessment. JAMA 265:1977-1981, 1991. 14. O'Leary DS: Editorial: Beyond generic screening. JAMA 265:1993-1994, 1991. 15. McGuire HH, et al: Measuring and managing quality of surgery: Statisticalvs incidental approaches. Arch Surg 127: 733-738, 1992.
,
Ronald L. Goldrnan, PhD, MPH Debby J. Walder, RN, MSN
An Initial Assessment of the Veterans Affairs Occurrence Screening Program
Identification o f poor-quality care remains a critical issue for hospitals and other health care organizations. Many organizations have continued to rely on certuin screening criteria thut are associated with preventable adverse events. Yet there remains little research on the sensitivity and specificity of such criteria. No one knows what the optimal set of screens is. This article by Goldman and Walder represents the kind of research that is leading us to more rational selection of screens. Relying on feedback from hospitals as well as a number of other independent analyses, the authors provide a good deal of insight into the utility of various screens. Their discussion of the costs of screening and the reliability of screeningjudgments is extremely useful. Although much more research is needed, Goldman and Walder appear to be moving us in the right direction. noyen A. Brennan, MD, JD, MPH, Professor of Law and Public Health, Harvard School of Public Health, and Associate Professor of Medicine, Harvard Medical School, Boston.
ccurrence screening has been defined a s the professional review of cases that involve adverse outcomes to identify opportunities for the improvement of care.' The Department of Veterans Affairs (VA) established an occurrence scmning program in its 159medical centers in October 1988;the program was designed to be consistent with the provisions of Public Law 100-322, which mandated the implementation of occurrence screening throughout the VA. This article will present data on the operation and effectiveness of the program and will describe recent modifications made in the program in response to these data.
Ronald L. Goldman, PhD, MPH, is Program Evaluator, Department of Veterans Affairs Central Office, Washington, DC. Debby J. Walder, RN, MSN, is Clinical Program Manager. Please address requests for reprints to Dr Goldrnan at the Office of Quality Management (15),Department of Veterans Affairs, 810 Vermont Avenue, NW, Washington, DC 20420. The authors wish to thank Thy Churchman for his work on data input and editing and three anonymous reviewers for their helpful comments.
Description of Original Program
Each medical center was required to apply the same nine screening criteria (Figure 1, p 328)to all acute care inpatients and to include the following four general stages in its program: 1. Screening to identify cases that meet the criteria; 2. Clinical review t o determine whether any aspects of these cases require further review; 3. Peer review to determine if practitioner care was less than optimal or, alternatively, review of system or nonpractitioner issues by an appropriate service or committee; and 4. Administrative review to determine the appropriate corrective action when peer review indicated that practitioner care was less than optimal. Within this general framework, medical centers selected the specific procedures so that the program could be adapted to local circumstances. VA policy developers attempted to minimize the resources required for the program by choosing screening criteria defined in terms of information avail-
able within the VA computer system. This allowed for automated identification of most cases requiring raiew, eliminating the need for manual review of medical records to identify occurrences, perhaps the most labor-intensiveaspect of other occurrence screeningprograms. Most exceptions to the criteria could not be automated and are identified through manual review.. To further reduce resource requirements, the developers established the clinical review function to minimize the amount of physician time needed to review cases. Data Collection
Monitoring of the implementation and effectiveness of the occurrence screening program has been continuous since its inception. The primary data collection instrument is a semiannual survey form that the quality management office at each'hospital completes after gaining input from other facility personnel and mails to the central office. Each report includes data on the results of the review process for each screening criterion, that is, the number of occurrences, the number of occurrences sent to peer review, and the peer review findings. The data for the period October 1989-March 1990were edited most intensively since staff were available at that time. This survey also included the most detailed questions about the implementationand effectiveness of the overall program and about the usefulness of each of the individual criteria. For these two reasons, data for that sixmonth period are presented here. The facility reports for that period were edited to identify the responses that were significantly different from those of comparably sized hospitals or from the data that the medical center had submitted for other six-monthperi-
Occurrence Screening Criteria Element 1. Readmission within 14 days'
2. Admission within 3 days following unscheduled ambulatory care visitt
3. Admission within 3 days following ambulatory surgery procedure* 4. Admission or absent sick in hospital from Veterans Affairs nursing home care unit* a. within 14 days of discharge from acute care b. to psychiatry service 5. Transfer from intermediate medicine* a. within 14 days of transfer from acute care b. to psychiatry service 6. Transfer to a special care units a. within 72 hours of transfer from a special care unit b. within 72 hours of a surgical procedure
7. Return to operating room in same admission11
8. Cardiac or respiratory arrest 9. Death#
Exceptions Readmission was scheduled at time of discharge Prior discharge against medical advice or irregular Scheduled admission Admission same day as visit Scheduled admission None
None
Planned transfer after surgery Transfer after emergency operation, which ordinarily requires transfer to special care unit Special care unit used as recovery room after surgery Planned transfer for procedure such as cardioversion Transfer from one special care unit to another Planned return to operating room documented prior to first surgery Two operations separated by more than 60 days Did nbt survive arrest Do-not-resuscitate (DNR) order or local equivalent at time of death Admitted for palliative care
'In a recent program revision criterion 1 was changed to readmission within 10 days. Two new exceptions are (1) readmission for alcohol or drug abuse, chemotherapy or radiation therapy; and (2) condition precipitating readmission did not exist at time of prior admission. t A new exception to criterion 2 was added: admission to Psychiatry. $Criteria 3, 4, and 5 were deleted. §Criterion 6 was modified to include only transfers to a special care unit within 72 hours of transfer from a special care unit. The exceptions are (1) planned transfer, and (2) transfer from one special care unit to another. IIA new exception to criterion 7 was added: second procedure unrelatedto first. #The first exception to criterion 9 was changed to DNR order at time of admission or more than 7 days prior to death.
Figure 1.Each medical center in the study was required to apply these nine occurrence screening criteria to all acute-care inpatients.
ods. We contacted the hospitals and asked to check their responses. A number of major errors were identified in this manner; these primarily concerned the number of occurrences identified. For 11hospitals it was not possible to correct inaccurate data retrospectively; we deleted these facilities from analyses involving the results of the review process (Figure 2, p 329). Two other hospitals had not sufficiently implemented
the program by October 1989to provide usable data for these analyses. Results Figure 2 shows the results of the review
process for the remaining 146facilities. Eighteen percent of the occurrences identified during the reporting period were sent to peer review. 'henty percent of the peer-reviewed occurrences received Level 2 or 3 ratings indicating
practitioner-related quality concerns. (See the practitioner quality-of-care scale in n b l e 1[p 3301 for definitions of Level 2 and 3 peer review ratings.)About 350 confirmed system/equipment issues were found. In all, opportunities for improvement related to practitioners or system/equipment issues were found in 4.2% of all occurrences.* There was considerable variability among medical centers. The proportion of occurrences referred to peer review was 8.6% for the 25th percentile and 212% for the 75th percentile. For the proportion of occmnces in which quality concerns were identified, the 25th and 75th percentiles were 1.7% and 7.9%, respectively We used a widely employed typology developed by Stefos et al to examine whether the results of the review process varied across types of medical centers. These authors employed cluster analysis to assign VA medical centers to one of six groups based on work load, scope of services, medical complexity of patients treated, and labor costs. The six types of hospitals are Group 1 (Small Affiliated), Group 2 (SmallGeneral),Group 3 (MidsizeAffiliated), Group 4 (Midsize General), Group 5 (LargeAffiliated),and Group 6 ( Psychiatri~).~ n b l e 2 ( p3:l) presents the findings of the analyses using these hospital groups. The overall variation among the six groups for both the proportion of occurrences in which opportunities for improvement were identified and the proportion referred to peer review was highly statistically significant (P = 0.0000),in part because of the very large sample sizes. As can be seen in %bled 2, the variation in identifying opportUnities for improvement was due to more frequent identification of such opportunities among Group 1 (Small Affiliated), Group 2 (Small General), and Group 6 (Psychiatric),the three groups with relatively small medical and surgical services and relatively few occurrences. The combined proportion for these three groups is 6.05% as com-
1
*The number O f occurrences at each facility was corrected fir theproportion of occurrences referred to peer review for which peer had not been completed when the report was submitted.
1
,
pared to a rate of 3.58% for the other three groups (X2 = 152.44, df = 1, P = 0.0000). The facilities in Groups 1, 2, and 6 also referred a larger proportion of occurrences for peer review (X2= 270.11, df = 1, P = 0.0000). However, this effect is largely due to the high rate shown by Group 1. To explore the perceived ektiveness of the program, we asked each facility how effective the occurrence screening program had been at identifying qualityof-care issues compared to other elements of their quality assurance (now termed quality management)program. The percentage of facilities that chose each of the five possible responses is listed in n b l e 3 ( p 331). 'Ib summarize briefly, 56% of the facilities selected "Most Effective Element" or "One of More Effective Elements," 24% chose "About Average," and 20% chose a response indicating the program had less-than-average effectiveness. The overall variation among the six hospital groups in the perceived effectiveness of occurrence screening was not statisticallysignificant at the 0.05 level; in particular, there was no tendency for Groups 1,2, and 6, the groups containing facilities with smaller medical and surgical programs, to evaluate the program more positively. Opportunities to improve care were attributed to the occurrence screening program by almost all of the facilities in response to a question on the survey asking for examples of such opportunities. Nearly one-half of the respondents described changes in policies and procedures that had resulted from the analysis of occurrence screening data. For example, several facilitiesdescribed changes in procedure intended to reduce the incidence of aspiration pneumonia among patients receiving enteral feedings.About one-third of the respondents described either interventions into the care of patients being reviewed or the development of improved methods for tracking the care of future patients, for example, procedures for the follow-up of suspiciouschest x-raysor critical laboratory findings. Another third of the facilities reported changes in medical record documentation. Staff education, especially pertaining to drug utilization, drug interactions,and drug toxicitx was
Results of Occurrence Screening Review Process for 146 Veterans Affairs Hospitals from October 1989 through March 1990 435,000 acute-care inpatient episodes
1 57,841 occurrences
10,698 referred to peer review ( 1 8.5%)
1 359 confirmed system1
9,412 completed peer reviews (88.0%)'
1 'The results of some peer reviews were not available when the reports were submitted by the hospitals.
Figure 2. This chart breaks down the results of the occurrence screening review process.
also mentioned by approximately onethird of the respondents. Enhanced staffing levels, the acquisition of new equipment, expansion of support servips, establishment of specialty arnbulatory care clinics, and development of clinical protocols were other examples frequently reported. Several facilities mentioned that additional equipment and staffing had been specifically dedicated to the monitoring of patients receiving sedation during procedures. Respondents also mentioned a number of times changes in the management of intravenous sites and lines to reduce the incidence of infections. Medical center responses also emphasized several general themes. A number of hospitals indicated that the program had increased the interest and involvement of physicians in peer review and that this would have a long-term positive impact on a wide range of quality management activities. Improvements in resident supervision and staff communication were also often attributed to the occurrence screening program. Effectiveness of Individual Screening Criteria Each facility was asked to rank the criteria with regard to usefulness in iden-
tifying opportunities to improve care. Since 3 of the 9 criteria involved two parts, there were, in effect, 12 criteria to rank. However, some criteria were not applicable to every hospital-for example, the criterion concerning returns to the operating room could not be used a t fa~ilities where surgery was not performed. n b l e 4 ( p 332) presents the proportion of facilities ranking each criterion among the three most useful. Readmissions ( # I ) ,death (#9),and admissions within three days of an unscheduled outpatient visit (#2),the three most effective criteria, were each ranked highly by 50%or more of the facilities. Return to the operating room ( #7) and cardiac or respiratory arrest (#8)were ranked among the top three criteria by a smaller, but still substantial, number of hospitals. Most of the remaining seven criteria were ranked as one of the three most useful criteria by no more than a few facilities. The large differences in the percentages of hospitals that ranked different criteria strongly indicate considerable agreement among the facilities with regard to the effectivenessof the 12 criteria. Not surprisingly,almost no differences were found between the ratings made by the six hospital groups
Table 1. Practitioner Quality-of-Care Scale
previously described; the major exception is that facilities in Group 6 (Psychiatric) gave relatively low ranks to return to the operating room ( #7) and return to a special care unit (#6a),criteria that have little relevance to them. A limitation of this analysis is that the appropriate interpretation of a criterion's rank as one of the three most effective by a hospital depends upon the total number of criteria used by that facility To control for this artifact, the rank each criterion received from a hospital was divided by the number of criteria used by that facility. We then computed the means of these adjusted ranks for each criterion. The results, which are shown in the last column of lkble 3, are almost identical to those found in the original analysis. Staffing Requirements of Program
The primary source of data on the staffing requirements of the occurrence screening program is a staffing survey designed to identify personnel whose primary responsibilitywas quality management. The VA's Office of Quality Management conducted this survey from April 1989 to July 1989, shortly after the inception of the occurrence screening program. One purpose of this survey was to ascertain the number of full-timeemployee equivalents( FTEEs ) a t each facility who were directly involved in specific quality management activities such as occurrence screening. For purposes of the survey, occurrence screening was defined in terms of the mandated screening criteria as well as any additional criteria the hospital had
have been a useful component of the VXs overall quality management effort at the time of data collection. The program, which was implemented throughout the might have handled system, resulted in the identification of large numbers of opportunities for improvement. Although not every facility had a good experience with occurrence screening, a majority of facilities regarded the program as one of the more effective elements in their overall quality management programs. 2. The staffing costs of the VA pro1 gram appear to be less than those 1 reported for occurrence screening pro- I grams in which manual screening is p e r f ~ r m e d . This ~ . ~ increased costeffectivenessis probably a consequence chosen to use. Only staff whose total of the use of computers for case identiinvolvement in quality management fication. Although it was initially diffiactivities represented 0.5 FTEE or more cult to develop software that met the were included in the survey Thus, the needs and expectations of all facilities, time of physicians and other staff who the use of computerized case identifireviewed records for the occurrence cation and data analysis has been a screening program, but who primarily highly positive aspect of the program. 7 performed clinical duties, was not These conclusionshave to be balanced j included. The median number of staff who worked in the occurrence screen- against several known important lirnitations of occurrence screening. Even ing program at the 128 hospitals with with the use of computerized screenfully implemented programs at the time of the staffing survey and that had com- ing, it is a resource-intensiveprogram plete$ the relevant portions of the staff- in which quality issues are identified ing survey was 0.7 FTEE. Seventy-two in only a small proportion of cases reviewed-4.2% in this report. In addipercent of these 128 facilities used 1.0 tion, peer reviey is central to occurrence or fewer staff in their program; 17% between 1.01 and 2.0 staff; and 11% screening; the available evidence indi- , cates that peer review, as usually pracmore than 2.0 staff. ticed, has quite limited interrater We also examined the relationship r e l i a b i l i t ~Of ~ . particular ~ relevance is I of the number of staff in a facility's prothe recent data reported by Rubin et a1 i gram to other variables. The number questioning the accuracy of peer reviews of occurrences, which indicates the work conducted by peer review organization load associated with the program, showed the strongest association (Spear- physician reviewers within the Health Care Financing Administration (HCFA) man rank correlation = 0.41, P = .~ 0.0001).There was also a significant rela- generic screening p r ~ g r a r n Third, ocamence screening programs typically tionship with the number of cases sent to peer review, another variable reflect- focus on deficient practitioner care,making work load (Spearman rank correla- ing it difficult to gain the enthusiastic involvement of clinical staff. FinaU3: the tion = 0.19, P = 0.04). There was no available data do not unequivocally supassociation with the measures unrelated port the effectiveness of screening crito work load, such as the percentage of teria in identifying cases for review that occurrences referred to peer review or are particularly likely to have quality the proportion in which opportunities concerns.69The recent finding of Rubin for improvement were identified. et al in which the HCFA generic screens were only slightly better than random Discussion sampling at targeting appropriate cases Several conclusions are suggested by for physician review is particularly notathe findings: ble in this regard.6 1. Occurrence screening appears to
'
1[ k
i P
$ i
t
The Office of Management has recently made a number of changes in the occurrence screening program to address the first three concerns. First, criteria 3 , 4 , 5 , and 6b have been dropped because of their relatively limited effectiveness;several of the remaining criteria have been refined to increase their yield. For example, the time period for the readmission criterion ( #l) has been changed from 14 days to 10 days; this modification is based on the recent finding of Ranki et a1 that almost all preventable readmissions occur within the shorter period.1° Similarly, the primary exception for the death criterion ( #9) has been changed from all do-notresuscitate (DNR)cases to only those in which the DNR order existed at the time of admission or more than 7 days prior to death. This change was based primarily on Dubois and Brook's finding that patients who had DNR orders written after admission often had preventable deaths." A second change is that annual evaluations of the interrater reliability of clinical and peer reviews are now being performedat each hospital. If agreement between reviewers is below a specified level, action to improve the reliability of the assessments is required. Finally, to expand the range of the program, the review process is now being used to identify exemplars:as well as deficient, practitioner care; greater emphasis is also being placed on the identification of opportunitiesfor improving care that involve system issues as recommended by Brooks et a1.12We are hopeful that these changes will enable the congressionally mandated VA program to overcome some of the usual shortcomings of the occurrence screening process. There are several limitations of this study. First, the data are from 1989and 1990; it is unclear to what extent they can be generalized to the present. However, analyses of more recent but less intensively edited data concerning the results of the review process (Figure 2) produced results highly similar to those reported here. Second, the data were self-reportedby the participating hospitals, and we could only edit them for major inaccuracies. After gaining input from other facility personnel, the survey forms were for the most part com-
Table 2. Results of Review Process for Six Groups of Veterans Affairs (VA) Medical Centers
Table 3. Responsesto Questions about Effectiveness of Occurrence Screening at Identifying Quality-of-Care Issues
pleted by quality management staff because they were the most knowledgeable sources of information. Their close involvement with the occurrence screening and other quality management programs may have biased their responses.* Finally, guidelines were not provided to the facilities to use in ranking the criteria regarding their usefulness in identifying opportunities for improvement nor were the facilities asked to explain their rankings. Thus, the data do not enable us to determine why some criteria were considered to be more *However, brief narmtive assessments o f the program by chiefs of stafi which were included in the same survey, were consistent with the ratings of quality management staff.
effective than others in generating quality improvements. Some commentators have suggested that occurrence screening should be largely replaced by methodologiesthat only involve peer assessment when a provider exceeds predetermined threshold, shows a negative trend, or is a statistical outlier in a comparativeanalysis.*." Since these approachesgenerally require less professional time, they may be more cost-effective. However, recent evidence suggests that occurrence screening procedures can be quite sensitive in identifying cases involving care that is significantlyproblematic. For example, Brennan et a1 showed that approximately 80% of malpractice claims involving medical causation of adverse
Table 4. Rankin
cations described in this paper. These studies should enable the VA to conclusively assess its occurrence screening program.
riieria
References 1. Goldman RL: Development of a Veterans Administration occurrence screening program. QRB 15:315-319, 1989. 2. Stefos T, LaVdee N, Holden F: Fairness in prospective payment: A clustering approach. Health Sen, Res 27:239-261,1992. 3. Craddick J W Improving Quality and Resource Management Through Medical Management Analysis. Rockville, MD: Medical Management Analysis International, Inc, 1987. 4. Bittle and Associates: Criteria Based Review for Integrating QA/RM/UR Programs. Baltimore, 1987. 5. Goldman RL: The reliability of peer assessments of quality of care. JAMA 266:958-960, 1992.
events or negligence could be identified through an occurrence screening review of the medical r e c ~ r dThus, .~ a well-run occurrence screening program should enable a hospital to identify many of its opportunities for improving patient care. The long-term role of occurrence screening and other methods involving review of individual cases will be in part determined by studies that show what proportion of these opportunities would also be uncovered through more statistically oriented approaches. In the only articlerelevant to this issue of which we are aware, McGuire et al reported that statistical analyses of surgical outcomes were less efktive in identifying patient care problems and in improving care than was individual case review for a 14-year period a t one facility.15 A second consideration is that the regular review of large numbers of individual cases in occurrence screening programs may indirectly improvequality simply because practitioners are aware that their care in any given case is likely to be reviewed. Thus, the costeffectiveness of occurrence screening may be quite different that the 4.2%
yield presented above suggests. The magnitude of the "surveillance effect" produced by different quality assessment methods is another important question for future research. Any definitive conclusion concerning the usefulness of occurrence screening can only be determined by studies that compare its cost-effectiveness to that of other quality management programs. Accordingly,we are examining the improvements in care that have resulted from occurrence screening and other VA quality management programs at a representative sample of facilities. The number and importance of the improvements in care resulting from each program will be related to the program's costs so that the costeffectiveness of the different programs can be compared. A second evaluation project will assess the sensitivity and specificity of peer review within the occurrence screening program. Next year, non-VA physician panels from the VAh new External Peer Review Program willindependentlyexamine 1,700 cases that had been previously peer reviewed. The VA is also planning to evaluate the impact of the recent program modifi-
6. Rubin HR, et al: Watching the doctorwatchers: How well do peer review organization methods detect hospital care quality problems? JAMA 267:2349-2354, 1992. 7. Barnes C, Moynihan C: Accuracy of generic screens in identifying quality problems: Analysis of false-positive and false-negativeo c c ~ c e slbpics . in Health Records Management 9:72-80,1988. 8. Brennan TA, et al: Identification of adverse events occurring during hospitalization. Ann In* Med ll2:221-226,1990. 9. Hannan EL, et al: A methodology for targeting hospital cases for quality of care record reviews. A m J Public Health 79: 430-436, 1989. 10. Ranki SE, Breeling JL, Goldman L: Preventability of emergent hospital readmission. A m J Med 90:667-674, 1991. 11. Dubois RW, Brook RH: Preventable deaths: Who, how often and why? An? Intern Med 109:582-589, 1988.
12. Brooks JHJ, et al: Systems versus performance problems: A peer review organization's perspective. QRB 18:172-177,1992. 13. Sanazaro PJ, Mills DH: A critique of the use of genericscreeningin quality assessment. JAMA 265:1977-1981, 1991. 14. O'Leary DS: Editorial: Beyond generic screening. JAMA 265:1993-1994, 1991. 15. McGuire HH, et al: Measuring and managing quality of surgery: Statisticalvs incidental approaches. Arch Surg 127: 733-738, 1992.