Control charts in hospital epidemiology and infection management: an update

Australian Infection Control

Control charts in hospital epidemiology and infection management: an update Anthony Morton

MSc(Appl), MD, MS Infection Management Services, Princess Alexanclra Hospital, Brisbane, QLD

Abstract Control chart methods were reviewed in Australian Infection Control in 20011. Since that time there have been improvements in these methods. A major advance has been the incorporation of risk-adjustment. In this article, what we regard to be the most useful control chart techniques currently available are described in a non-technical way. For those who wish to implement these methods, suitable nonmathematical references are included.

Introduction

Numerical data

There are broadly two types of charts used to signal a real increase in adverse events (AEs); those that are used on data collected periodically, for example annually, from a number of hospitals (between-institution charts) and those that are used to analyse sequential data in a single institution (within-institution charts).

Antibiotic usage is numerical data where the numerator would be the number of DDDs of the drug prescribed in the period of interest and the denominator would be the number of occupied bed-days for that period.

In addition, charts are used with binary data (for example surgical site infection (SS!) data or postoperative deaths where an event does or does not occur), count data (such as bacteraemias, needlestick injuries, medication errors and infections and colonisations with multiple antibiotic resistant (MRO) organisms) and numerical data (such as the number of defined daily doses (DDDs) of an antibiotic prescribed in a month).

Limitations

Binary data Within-institution binary data are usually analysed in sequence with each patient given a score of one if the AE occurs and zero if it does not occur; if there is risk-adjustment, an additional figure of the probability of that patient having that AE is also available. When the data are very numerous and the proportion of patients having the event is above about 10%, such as mortality in a busy intensive care unit (ICU), the data may be amalgamated, for example by months (with the numerator being the number of AEs such as deaths within 30 days of patients admitted to the ICU in that month and the denominator being the total number of relevant patients such as the number admitted to the ICU in that month). With risk-adjustment, there is the additional figure of the expected number of deaths in those patients.

Count data With count data, the numerator is usually the number of AEs occurring in a month and the denominator is usually occupied bed":days or device-days for device-related AEs such as central line related bacteraemias.

Between-institution charts Comparisons between institutions has been an area of controversy. In a related paper in this issue of Australian Infection Control, we express our belief that it is better to move towards the use of withininstitution charts. A typical chart displaying between-institution data would present amalgamated annual data for each hospital within a group of hospitals. A limitation of this type of analysiS is that a run of unsatisfactory outcomes within an institution can easily be missed if the data for that institution are otherwise satisfactory. In this instance, the staff of such an institution would like to have known when an unsatisfactory run occurred so that corrective action could have been taken. Another disadvantage is that, even if a signal occurs, by the time the charts are produced and disseminated, the cause may have passed and it may be impossible to detect it. In addition, between -institution data displays encourage comparisons of institutions. Such comparisons are unhelpful as differences can easily be due to random variation, for which correction is difficult, or be due to demographic differences that are-difficult to deal with using risk-adjustment. Finally, between-institution analyses often employ routinely collected data that may be of poor quality, that may not be collected primarily to study the AE of interest or that may be gamed in some way to make them look better 2. In spite of these limitations, central authorities require betweeninstitution analysis of AE data. It is therefore important that the Vol 11 Issue 1 March 2006

methods used give as accurate a picture as possible, using data that are appropriate for the analysis.

Funnel plots In the past, league tables have been used for between -hospital data analysis. This approach invites unhelpful comparisons; Adab and colleagues proposed that con trol chart methods be employed ' . These are better able to control for the effects of random variation and are less Ukely than league tables to invite incorrect comparisons. Professor David Spiegelhalter of the UK Medical Research Council in Cambridge has described a funnel plot that is currently the preferred method for the display of these data". Figure 1 shows some data on monthl y au topsy ra tes analysed in a standard Shewhart control chart, with tht; number of deaths used as the denominator and the number of autopsies performed as the numerator. We described this type of chart in our previous

Australian Infection Control paper I and in a related paper s. You can see that, in some months, the upper control limit is higher or lower than the previous months. When the number of deaths in the hospital for the month is low, the control Umits are wide and th ey are narrow when that number is large. Since these data are a time-series, the months are represented on the horizontal axis and have to remain in chronological order. However, if the horizontal axis showed hospitals for which data on deaths and autopsies were being analysed, the data could be ordered by size of hospital. So, in a funnel plot, the hospitals are sorted by their size with the smallest on the left. Since the control limits are determined by the sizes of the samples for each of the hospitals, they will resemble a funnel. This is illustrated by some bacteraemia data from a number of Level 2 and 3 hospitals in Figure 2. This is an excellent chart for visual representation but there remai n three further issues that must be addressed; risk-ad justment, overdispersion and employment of shrinkage methods.

Risk adjustment Risk-adj ustment for SSIs wiU be familiar to readers, although there is a need for methods that are better suited to Australian publiC hospitals, as identified in aggregated Australian data 6. When funnel plots are used with risk-adj usted data, the data are standardised indirectly. The rate for each hospital is divided by th e expected rate for that hospital to get a standardised morbidity or mOltality ratio (SMR) . The SMRs are then multiplied by the overall rate to obtain a riskadjusted rate. These are then used as the data for th e funnel plot in the usual way; the midline of the chart is the weighted average of the risk-adjusted rates and approximate control limits are calculated by assuming that these rates have, at least approximately, a binomial or Poisson distribution. The method is described fo r binary data by Hart and colleagues 7, and for count data by Spiegelhalter 8 . Vol 11

Issue 1

March 2006

-

Multiple risk!

Australian Infection Control Figure 3 shows bacteraemia data for Level 1, 2 and 3 hospitals combined. Since the Level 1 hospitals have oncology-haematology and other units that perform invasive procedures, their bacteraemia

is then the observed rate for hospital A diVided by its expected rate. This is now multiplied by the average rate for all the hospitals to get the risk-adjusted rate for hospital A This is now used in the funnel

rates can be expected to differ from those of Level 2 and 3 hospitals.

plot as described above. Figure 4 shows the result; the internal dotted red line is the resulting control limit.

Figure 3 shows that nearly half of all the hospitals'bacteraemia rates are outside the control limits. To risk-adjust the data, for example for Level 1 hospitals, the average rate for Level 1 hospitals is calculated

by summing all the Level 1 numerators and dividing by the sum of their denominators. For hospital A the expected rate is then determined by multiplying this average by the hospital A denominator. The SMR for hospital A 0.51--------------::======:::::--~ 0.4 ....

0,3

0,2 0.1 0.0

---

~---.---.---.---.---.---.---.---,---,---,---.~

May '85

Mar '85

Jul '85

Nov'S5

Sep '85

Blue = Data Red =1, 2 & 3 SD IimiL'

Figure 1. Monthly autopsy rate January 1985 - December 1985. 1, 0 - . - - - - - - - - - - - - - - - - - - - - - - - - - - ,

0,6 o

0.4

o

__----------~---------o---­ -----------------0.0 '--;-----,----r---.--.---.--- .--- .---,----,----,---,--' o

20

17

19

15

residual heterogeneity or excessive variability (overdispersion) will usually remain. Spiegelhalter S has described two simple methods for dealing with this. The result of applying the first of the methods described in his paper has resulted in the outer solid red lines in Figure 4. These are the final two standard deviation control limits

9

Shrinkage methods A difficulty that exists, especially with small hospitals, is that confidence limits and control limits can be very imprecise. In a judgmental environment this can lead to hospitals being penalised for apparently poor results that are in reality due to random variation. A method for dealing with this is Bayesian Shrinkage described by Gibberd and colleagues 9. This is an advanced concept and at present software for its implementation is limited; however, its use has the support of some central authorities in Australia. Basically the idea is to assume that the rates for the hospitals have their own distribution and this makes it possible to 'borrow' from the overall distribution of all the hospitals' data to make the predicted values for individual hospitals more precise. The downside is that

0,8

0.2

In spite of efforts to risk-adjust data, demographic differences between hospitals will not be adjusted for completely and some

for that funnel plot.

z . ... . . . . ~ . . .. ~ ran '85

Overdispersion

23

12

Hospital n umbers

6

3

18

2

Blue = Data Red 2 SD limits

it may also make their predicted values more biased. This means that false positive results become much less likely (a very important objective in a judgmental environment) but it may be more difficult to detect genuine differences. Thus when shrinkage is used, it is probably wise to display the results of the analysis both with and without it. A non-technical discussion can be found in Christiansen and Morris 10. Gibberd and colleagues discourage comparisons; they

=

Figure 2. Bacteraemias Level 2 & 3 hospitals: funnel plot. 1.5 -.-------------------------------------------------,

1.5,----------------------------------------------.

---

1,0

1.0

0

0

0--0.....- - - -- - __

a 9

0

0.5 0

~.

0

0

--

0.5

-----

__ ___ ____

---- ----o

"

"

-- A --- ~ ----- - ·

o

0

0.0 L-.,---.---.----.__.---.---.-----,__,----,----,-__.---.---.----.__.---.-----.---' 20 17 4 19 15 9 5 23 12 18 3 1 6 13 8 2 7 10 Hospital numbers Blue = Data Red = 2 SD limits

Figure 3. Bacteraemias all hospitals: funnel plot.

20 17 4

19 15 9

5

23 12 18

3

Hospital numbers

6

13

8

2

7

10

Blue = Data Solid Red = Dispersion adj u"red 2 SD limits Das hed Red = Dispersion unadjusred 2 SD limit,;

Figure 4. Risk-adjusted overdispersion corrected bacteraemias all hospitals: funnel plot. Volll

Issue 1 March 2006

Austra lian Infecti on Con trol poin t out tha t it is more prod uc ti ve to atte m pt to m ove the w ho le o utcome dis tributio n so th a t less AEs occur. Unfo rtun a tely, it ca n be very d iffic ult to stop people makin g com pa riso ns o n data th a t

3

are sub ject to ra nd o m va riatio n a n d di ffering risk,

Within-institution charts

2

The odds CUSUM A powerful new CUSU}"l is ava ilable fo r ana lys ing binary data deals

\~~th

1I 12 .

risk-adjusted da ta and can be extended to co unt data I J

It I.,

Implemen ta ti on is d escri bed in Beiles & Morton l ' .

0

A grea t advantage is that da ta on simi la r operations perform ed by

Jan '03

May '03

~ Sep '03

Jan '04

N\

Jun '04

Sep '04

Dec '04

Apr '05

Aug No" '05

'05

the same surgi ca l tea m with d iffering expected SSI ra tes can be ama lgamated . For exam ple, there may be insu ffi cien t nu mbers o f bowel excisions with colostomy for analysis to detec t problems in a timely m anner. H owever, th ese data may be ilma lgil m ated wit h bowel excisio ns wi th out colostomy. H aving a t least 100 operJ tions per year is idea l bu t w hen the re are less th a n abo ut 50, changes in outcome ra tes may take too long for CUSUl\·1 ana lysis to detect the m pro m ptly. This CUSUM is shown in

Fi ~;u re

5. There were

Figure 5. SSls following bowel eXCISIOn with and without c%stomy CUSUM test chart !alllwnj 2003 - December 2005 (n == 4.31). num ber exceeded 400. The expected rate for in -hospital SSls w hen colostomy was perform ed was esti mated at 5% and 3% \·vitho ut colostomy.

just over 100 bowel excisions with colostomy in almos t 3 yea rs

Thi s chart has two d isadva ntages. First, although good for de tectin g

but, when grouped with bowel excisions withou t colostomy, th e

runs of AEs, it gives no in forma ti on about the overall seri es

First Choice in ... CERDAK ™

Vol11

Issue 1

March 2006

r) •

010 C

e (a rrl le:; •

Australian Infection Control of outcomes and must be supplem nted by another chart like the cumulative observed minus expected (cumulative O -E) chart

(there were about four less AEs than expected at that point) to a cumulative excess of approximately two AEs at the end of the series.

described below. Second, when monitoring outcomes like cardiac

With an overall excess of two SSIs in over 400 operations, the lower

surgical deaths that have a very low overall expected rate but some

credible limit does not exceed zero as it would if the overall number

patients have a poor prognosis, it can give false positive signals if a surgeon operates on a large number of th e latter patients. This is unlikely to cause problems with SSIs.

of SSIs was excessive. However, the chart provides a good overall

Cumulative O-E chart The cumulative O -E chart provides a good summary of an overall

display of the data. The CUSUM chart (Figure 5) provides no overall information but it 'signals' as a result of the run of SSIs near the end of the series.

Shewhart/EWMA chart

series of binary outcomes. It can also be used with count data and

We believe that the cumulative O-E chart for count data and the

can be a useful tool for infection control practitioners to provide

corresponding CUSUM chart for count data Il are most suitable for

feedback to ward staff and to show administrators the extent of any

relatively uncommon outcomes such as MRSA bacteraernias. For

problem. For example, if historically about one MRSA bacteraemia is expected per month in a hospital, the demonstration that there are now four or five more than expected in a cumulative O -E chart

most count data of AEs such as monthly bacteraernias and MRO colonisations, we prefer to employ the ShewhartfEWMA chart that we described in our previous AlC paper I .

can be an effective way to present the problem to the hospital administration . The cumulative O -E chart has its origins in the VLAD chart made popular in the UK for monitoring cardiac surgical outcomes 16. The VLAD chart is a cumulative expected minus observed chart and the cumulative O-E chart is thus its mirror image. We find that staff naturally associate increased numbers of AEs with an upward deflection in a chart and thus we believe that the cumulative O -E version is superior. Control limits for this chart were proposed by Sherlaw-Johnson and colleagues 17, 18. We prefer to employ Bayesian updating 19. This gives credible limits that are similar numerically to confidence limits but that have a more natural interpretation (the method is described briefly in the appendix) .

The major recent advance with Shewhart/EWMA charts is the ability to incorporate risk-adjustment 7. The method is described in the earlier section on funnel plots. It is currently most useful when there are large datasets of risk-adjusted binary data having expected outcome rates exceeding about 10%, such as mortality in large ICUs that typically have average mortality rates around 15%. The chart is illustrated in Figure 7. Another important issue with count data hospital acquired infections is the independence of observations that is clearly violated when there is transmission of an MRO. This can cause clustering and increased variability; we have dealt with this by using the negative binomial distribution rather than the usual Poisson distribution to obtain control limits 5. However, the methods for dealing with

Figure 6 is the cumulative O - E chart that corresponds to the CUSUM chart in Figure 5. Figure 6 shows a run of SSIs on the right where the cumulative excess number goes from about -4

excess variability between hospitals described by Spiegelhalter 8 and referred to in the earlier section on funnel plots may prove to be superior.

0.30.-- --

- - -- - - - -- - -- - --

- -- - - - - ,

0.25

10

0.20 5

0.15

O~--------~--------~--------------~----~~

0.10 0.05

-5

0.00 -r,-rT<- .. .---,-,--,- rT<- .. .---'-rT-,--rr-,,-rT<-.. .---'-rT.-1

Ja n '95

JM

'03

May '03

Sep '03

Jan '04

Jun

'04

Sep ' 04

Dec '04

Apr '05

Aug Nov

'05

'OS

(Tick marks eve!), 50 units)

Figure 6, SSJs following bowel excision with and without colostomy O-E chart January 2003 - December 2005 (total 431).

C;r,'~n

Jun '95 [),It.l

Nov'95

Apr '96

Sep '96

Feb '97

Jul '97

Dec '97

& 2 SD 11'1111,

Blue = EWMA li ne Red = 1:.1VMA 2.5 SD limit,

Figure 7. Risk-adjusted JeU mortality January 1995 - December 1997: comparison with standard chart,

Vol 11 Issue 1 March 2006

Australian Infection Control

Other charts

References

We find that charting of monthly antibiotic usage is valuable. We

1.

Morton A & Curtis M. Control charts for nosocomial infection surveillance. Aust Infect Cont 2001; 6:61-65.

2.

Pitches D, Burls A & Fry-Smith A. How to make a silk purse from a sow's ear - a comprehensive review of strategies to optimise data for corrupt managers and incompetent clinicians. BM) 2003; 327:1436-

described an approach in a previous Australian Infection Control paper 20. These data are typically highly variable and we now employ only an EWMA chart, frequently with a large weight of 0.3 to 0.5 as this improves the chart's ability to display trends. Finally, there is the issue of monitoring MRO burden, for

1439. exampl~

daily prevalence of these organisms in the hospital or its wards. These data display marked autocorrelation so that today's MRO burden is most likely to be predicted by yesterday's value; the challenge is to be able to detect a breaking out from this pattern.

3.

Adab P, Rouse A, Mohammad M & Marshall T. Performance league tables: the NHS deserves better. BM) 2002; 324:95-98.

4.

Spiegelhalter D. Funnel plots for institutional comparisons. Qual Saf Health Care 2002; 11:390-391.

5.

Morton A, Whitby M, McLaws M-L, Dobson A, McElwain S, Looke D, Stackelroth ) & Sartor A. The application of statistical process control charts to the detection and monitoring of hospital-acquired infections. ) Qual Clin Prac 2001; 21:112-117.

6.

McLaws M-L & Taylor P. The hospital infection standardised surveillance (HISS) program: analysis of a two-year pilot. ) Hosp Infect

Conventional control charts should not be used with data that display marked auto correlation, so alternative approaches need to be found. modelling 21.

The most promising of these is mathematical

The factors aiding transmission such as overcrowding, MRO burden, antibiotic misuse, poor hygiene and hand washing practices, device

2003; 53:260-268.

7.

Hart M, Lee K, Hart R & Robertson W. Application of attribute control charts to risk-adjusted data for monitoring & improving health care performance. Qual Man Health Care 2003; 12:5-19.

8.

Spiegelhalter D. Handling over-dispersion of performance indicators. Qual Saf Health Care 2005; 14:347-351.

9.

Gibberd K Parthmeswaran A & Burtenshaw K. Using clinical indicators to identify areas for quality improvement. ) Qual Clin Prac

10.

Christiansen C & Morris C. Improving the statistical approach to health care provider profiling. Ann Intern Med 1997; 127:764-768.

11.

Steiner S, Cook R & Farewell V. Risk adjusted monitoring of binary surgical outcomes. Med Decis Making 2001; 21:163-169.

12.

Steiner S, Cook R, Farewell V & Treasure T. Monitoring surgical performance using risk adjusted cumulative sum charts. Biostatistics

misuse and inability to detect MRO carriers promptly and discharge or isolate them efficiently are interdependent - employing a mathematical model allows them to be studied together. We must encourage research using this tool.

Appendix: Bayesian updating For binary data, the series starts with a beta (1,1) prior and for count data a gamma (1,.001) prior. Then, for each new binary outcome, one is added to the first beta parameter if an AE occurs and one is added to the second parameter if no AE occurs. Credible limits are then found with the quantile function of the beta distribution.

2000; 20:136-144.

These must be multiplied by the number in the series to that point. There are then the cumulative observed number of AEs and its

2001; 2:441-452.

°

13.

Spiegelhalter D, Kinsman R, Grigg & Treasure T. Sequential probability ratio tests for monitoring risk -adjusted outcomes. Int) Qual Health Care 2003; 15:7-13.

the cumulative O-E chart.

14.

With count data, the monthly count is added to the previous

Morton A. The use of statistical process control methods in monitoring clinical performance. Int) Qual Health Care 2003;15:361-362.

15.

Beiles B & Morton A. Cumulative sum control charts for assessing performance in arterial surgery. Aust NZ) Surg 2004; 74:146-151.

16.

Lovegrove J, Valencia 0, Treasure T, Sherlaw-Johnson C & Gallivan S. Monitoring the results of cardiac surgery by variable life-adjusted display. Lancet 1997; 350:1128-1130.

17.

Sherlaw-Johnson C. Lovegrove J, Treasure T & Gallivan S. Likely variations in perioperative mortality associated with cardiac surgery: when does high mortality reflect bad practice. Heart 2000; 84:79-82.

18.

Sherlaw-)ohnson C. Morton A, Robinson M & Hall A. Real-time monitoring of coronary care mortality: a comparison and combination of two monitoring tools. Int) Cardiol 2005; 100:301-307.

19.

Resnic F, Zou K, Do D, Apostolakis G & Ohno-Machado L. Exploration of a Bayesian updating methodology to monitor the safety of interventional cardiovascular procedures. Med Decis Making 2004;

upper and lower credible limits. The corresponding cumulative expected number of AEs is then subtracted from each of these to get

cumulative monthly count to give a current cumulative value 'm' and its credible limits are found with the quantile function for gamma(m+1,1.001). The corresponding cumulative expected values are then subtracted from these values. This count data method is best reserved for uncommon AEs, for example those having expected monthly counts of less than two, as the Shewhart/EWMA chart is less easy to interpret when rates are so low. Credible limits are similar numerically to confidence limits but their interpretation is that, for 95% limits, the probability is 0.95 that the true value lies between the limits. The definition of a 95 % confidence limit is that, if an experiment is repeated 100 times, the true value will lie within the limits on 95 occasions. This latter idea is difficult to envisage with surveillance data where the sample analysed is often the entire available sample for the hospital of interest and there is no large population from which repeated random samples could have been taken. Vol 11 Issue 1 March 2006

24:399-407. 20.

Morton A & Looke D. Charts for surveillance of antibiotic usage. Aust Infect Cont 2003; 8:14-19.

21.

Grundmann H & Hellriegel B. Mathematical modeling: a tool for hospital infection control. Lancet Infect Dis 2006; 6:39-45.