- Email: [email protected]
16 Audit
16 Audit R. J. L I L F O R D
Audit means different things to different people and one's view of the subject depends on which definition is selected. Obstetricians are quick to take credit for instituting audit in the form of local and national data collection exercises, such as statistics on perinatal mortality, birthweight etc. While these exercises certainly constitute observational studies, they cannot be used to make firm conclusions about the quality of care. The latter is probably one of the less important variables affecting outcomes such as perinatal mortality and neurological morbidity. Thus, any argument for a change in resourcing or methods of care, predicated on these statistics, is likely to be specious. COMPUTERS AND AUDIT It is often claimed that the proliferation of computer systems, such as those used to collect the Korner maternity data-set, contribute to audit. This argument is based on the notion that these more detailed data-sets contain sufficient information to allow researchers to control for other variables, such as social class. This idea is fallacious for two reasons. Firstly, these data-sets are often inadequately detailed 'minimal data-sets' and this rationalization appears necessary in order to ensure the quality of retrospective data at a national level. Secondly, and far more importantly, statistical techniques, such as multivariate analysis and logistic regression, can only control for known sources of bias and unknown factors are likely to be more important, especially in a socially and economically sensitive subject such as perinatology. There is, therefore, no sound inference that can be made from a review of information contained in amalgamated databases of hospital statistics. Audit, as properly defined, hinges on inference--the inference that the quality of care was or was not of a high standard. Descriptive statistics, therefore, can be used to generate hypotheses but should not be used as a form of audit. ANALYSIS OF CASES OF POOR OUTCOME Rather different to the anonymous data collection mentioned above are Bailti~re' s Clinical Obstetrics and Gynaecology--
Vol. 4, No. 4, December 1990 ISBN 0-7020-1479-6
905 Copyright © 1990, by Bailli~re Tindall All rights of reproduction in any form reserved
906
R . J . LILFORD
those national surveys which not only measure overall mortality and morbidity, but which also examine in detail each case with a bad outcome. Here, experts make an informed, albeit subjective, judgement as to whether the bad outcome could have been prevented by a higher standard of care. This process, therefore, is similar to cases of civil tort, where the Court must decide about negligence and causality on the balance of probability. Unlike a court of law, this form of audit is designed to warn other doctors about problems and pitfalls (e.g. be careful to count your swabs during episiotomy repair). Such reports make compelling reading for doctors and are much like the yearly reports published by the defence societies. While thoroughly supporting these activities and finding them very informative in warning about pitfalls which lie ahead, a minor note of caution must be sounded. These are one-sided audits, in which only those cases with a bad outcome are examined. This, by its very nature, is likely to bias an assessor in the direction of criticizing clinical care. An assessor or reader must therefore avoid the superficially plausible question: 'Would the bad outcome have been avoided by a different form of care?' The answer will often be 'in all probability--yes', but it does not follow from this that the care was of a poor quality. The fact that aparticular patient would have most likely been spared a bad outcome by a different form of care does not mean that other such patients should receive this treatment--probabilistically it may have had a lower chance of success. In decision theory parlance it may have had a lower expected utility. One possible method of avoiding such subtle biases which are likely with non-blinded one-sided audit is to take random controls with similar presenting features and examine the methods of care in these patients. Of course, if no difference is found, this leaves open the question of whether care was generally poor, as opposed to poor only in those patients with a bad outcome, DEFINING THE APPROPRIATE STANDARD
The above consideration takes us straight into discussion of how we decide on the appropriate standards of care. These standards must be those that maximize the chances of a good outcome (we deal here with the simple case where there are no trade-offs between various outcomes). The first point to make is that auditing the quality of care involves a study of process. It therefore depends on the assumption that we know which practices maximize beneficial outcomes. This exercise is therefore relevant only when we have good evidence linking the process of care with these outcomes. Judgement about whether particular clinical or management policies are more likely to do good than harm will be based on different forms of evidence. At one extreme, the effects of policies may be obvious, for example the beneficial effect of blood transfusion after massive haemorrhage, the drainage of wound abscess or the prescription of antibiotics for life-threatening infection. The trouble is that the more obvious the benefit of the medical intervention, the less likely are doctors to ignore it. Thus, assessment of the
AUDIT
907
quality of medical care in deaths from massive post-partum haemorrhage usually hinges on whether blood transfusion was adequate or excessive, a somewhat more subtle and contentious point than failure to establish an intravenous line! There is, however, one area where clear guidelines can be laid down and where, in my view, audit has its greatest role. This is in the quality of administrative arrangements. Many of the appropriate processes are simply good manners but I believe these can be effectively audited. Thus audit can include such factors as 'were the general practitioner and midwife informed of a perinatal death before the patient's discharge?' or 'Have doctors recognized and acted upon an abnormal smear result?' It is my view that it is in these organizational matters that the health service most often falls down, and it is here that audit has the most to offer. In the above examples the accepted standard against which the process of care can be compared is very obvious. In other cases, however, the accepted standard should itself be audited to ensure that it is based on sound evidence. Chalmers (I. Chalmers, personal communication) pointed out the fundamental flaw in any form of audit that is restricted to an examination of the process of health care by citing an early example of medical audit. This involved censure of a Boston physician who had not prescribed the then standard treatment for pneumonia--blood letting. It is important that audit does not become an authoritarian tool to restrict the advance of medical knowledge. Thus, any requirement of particular policies for care or management within the health services should be based on good evidence that these policies are more likely to do good than harm. Evidence about the effects of care may take a variety of forms, including seeking the views of people who have used the health service. Here again caution is needed since we do not know how prior expectations among different groups of people influence their reactions to the health service. Other methods of care have been evaluated in properly controlled studies, thus generating evidence required for informed audit of practice. There is now good evidence that certain forms of care are more effective than others (for example, catgut sutures cause more pain following perineal wound repair, steroids promote fetal lung maturity and prophylactic antibiotics reduce the morbidity and financial costs of caesarean section, while oxytocic agents reduce the incidence of post-partum haemorrhage) (reviewed in Chalmers et al, 1989). However, many practices which are enshrined in obstetric dogma are of completely unproven value and may even be harmful; a good example is the glucose tolerance test. We therefore have good evidence on which to base certain policies and these can be used as a 'gold standard' against which the process of care can be judged. Other examples are much more tenuous and, before resorting to audit, more properly controlled, randomized experiments should be carried out. AUDIT OF AUDIT Audit itself needs to be audited. A good example of this is Barrett's caesarean
908
R.J.
LILFORD
section audit (Barrett et al, 1989) in which five auditors reviewed the case notes of a random sample of emergency caesarean sections for fetal distress. The initial audit showed a wide inter-observer variation, all five auditors agreeing in only 20% of cases. Furthermore, when re-presented with the same case notes, a marked inconsistency with first-time decisions was observed, the cumulative inconsistency rate being one case in four. This study reinforces the point that audit must be preceded by a validation of the audit process itself.
SUBTLE EFFECTS OF AUDIT There is one final value of audit which lies beyond the more straightforward benefits that may result from the direct educational and disciplining effects. This is a more nebulous but ironically possibly more important effect which results not from the information provided by audit, but from the very process of audit itself. This is the famous Hawthorne effect, whereby the very process of examining a human activity, improves that activity. Thus audit has a symbolic component to it: it sends a signal to health professionals that their work is being monitored and that they are accountable to their patients. In a rather more parochial British sense, audit forms an important component of the proposed health service reforms. The essence of these is the separation of responsibility for providing and purchasing medical services and the establishment of an internal market. However, many services, and antenatal care is an example, are semi-monopolies. The provider of these services might be in too strong a bargaining position for funds (cost per patient) if there was no link between remuneration and some attempt to measure the value of service in terms of outcome and consumer satisfaction.
ON-LINE AUDIT Lastly, audit is a retrospective process of quality control. It has educational and disciplining value and is designed to lower the chance that deficiencies or shortcomings will be repeated. The individual patient who may have had an increased risk of poor outcome as a result of the care standard which has been audited and found wanting will not benefit. It is for this reason that computerized history-taking and data transfer systems have great advantages in clinical care. They both educate and warn of pending risk in a particular patient prospectively rather than retrospectively. 1. 2. 3.
Hospital information systems reduce the risk of missing data (Chapter 3) by making laboratory or imaging data readily available. Pharmacy systems warn of drug interactions. Branching history systems ensure complete relevant data collection (Chapter 5).
AUDIT
4.
5. 6.
909
Low level artificial intelligence (action suggestions) remind the clinicians of appropriate tests and risks, given that humans do not have unlimited powers of recall and concentration and operate in an increasingly complex environment (Chapters 5, 10 and 12). More advanced artificial intelligence and Bayesian systems improve diagnostic precision in selected instances (Chapters 11 and 14). Electronic interfacing (Chapter 8) results in improved recognition of abnormal electrophysiological patterns (Chapter 9).
In all these cases the computer is used in the care of a particular patient, so that potential errors are covered in advance and patients are protected. Such systems also produce data for retrospective research and indeed they can be used to automate trials (Chapter 7) and thereby define better the appropriate standards of care for future on-line and retrospective audit. REFERENCES Barrett JFR, Jarvis GJ, Mcdonald HN, Buchan PC, Tyrrell S & Lilford RJ (1990) The inconsistencies o f clinical decisions revealed by an obstetric audit. Lancet (in press). Chalmers I, Enkin M & Keirse MJMC (1989) Effective care in pregnancy and childbirth: a synopsis for guiding practice and research. In Chalmers I, Enkin M & Keirse MJMC (eds) Effective Care in Pregnancy and Childbirth. Oxford: Oxford University Press.
Audit means different things to different people and one's view of the subject depends on which definition is selected. Obstetricians are quick to take credit for instituting audit in the form of local and national data collection exercises, such as statistics on perinatal mortality, birthweight etc. While these exercises certainly constitute observational studies, they cannot be used to make firm conclusions about the quality of care. The latter is probably one of the less important variables affecting outcomes such as perinatal mortality and neurological morbidity. Thus, any argument for a change in resourcing or methods of care, predicated on these statistics, is likely to be specious. COMPUTERS AND AUDIT It is often claimed that the proliferation of computer systems, such as those used to collect the Korner maternity data-set, contribute to audit. This argument is based on the notion that these more detailed data-sets contain sufficient information to allow researchers to control for other variables, such as social class. This idea is fallacious for two reasons. Firstly, these data-sets are often inadequately detailed 'minimal data-sets' and this rationalization appears necessary in order to ensure the quality of retrospective data at a national level. Secondly, and far more importantly, statistical techniques, such as multivariate analysis and logistic regression, can only control for known sources of bias and unknown factors are likely to be more important, especially in a socially and economically sensitive subject such as perinatology. There is, therefore, no sound inference that can be made from a review of information contained in amalgamated databases of hospital statistics. Audit, as properly defined, hinges on inference--the inference that the quality of care was or was not of a high standard. Descriptive statistics, therefore, can be used to generate hypotheses but should not be used as a form of audit. ANALYSIS OF CASES OF POOR OUTCOME Rather different to the anonymous data collection mentioned above are Bailti~re' s Clinical Obstetrics and Gynaecology--
Vol. 4, No. 4, December 1990 ISBN 0-7020-1479-6
905 Copyright © 1990, by Bailli~re Tindall All rights of reproduction in any form reserved
906
R . J . LILFORD
those national surveys which not only measure overall mortality and morbidity, but which also examine in detail each case with a bad outcome. Here, experts make an informed, albeit subjective, judgement as to whether the bad outcome could have been prevented by a higher standard of care. This process, therefore, is similar to cases of civil tort, where the Court must decide about negligence and causality on the balance of probability. Unlike a court of law, this form of audit is designed to warn other doctors about problems and pitfalls (e.g. be careful to count your swabs during episiotomy repair). Such reports make compelling reading for doctors and are much like the yearly reports published by the defence societies. While thoroughly supporting these activities and finding them very informative in warning about pitfalls which lie ahead, a minor note of caution must be sounded. These are one-sided audits, in which only those cases with a bad outcome are examined. This, by its very nature, is likely to bias an assessor in the direction of criticizing clinical care. An assessor or reader must therefore avoid the superficially plausible question: 'Would the bad outcome have been avoided by a different form of care?' The answer will often be 'in all probability--yes', but it does not follow from this that the care was of a poor quality. The fact that aparticular patient would have most likely been spared a bad outcome by a different form of care does not mean that other such patients should receive this treatment--probabilistically it may have had a lower chance of success. In decision theory parlance it may have had a lower expected utility. One possible method of avoiding such subtle biases which are likely with non-blinded one-sided audit is to take random controls with similar presenting features and examine the methods of care in these patients. Of course, if no difference is found, this leaves open the question of whether care was generally poor, as opposed to poor only in those patients with a bad outcome, DEFINING THE APPROPRIATE STANDARD
The above consideration takes us straight into discussion of how we decide on the appropriate standards of care. These standards must be those that maximize the chances of a good outcome (we deal here with the simple case where there are no trade-offs between various outcomes). The first point to make is that auditing the quality of care involves a study of process. It therefore depends on the assumption that we know which practices maximize beneficial outcomes. This exercise is therefore relevant only when we have good evidence linking the process of care with these outcomes. Judgement about whether particular clinical or management policies are more likely to do good than harm will be based on different forms of evidence. At one extreme, the effects of policies may be obvious, for example the beneficial effect of blood transfusion after massive haemorrhage, the drainage of wound abscess or the prescription of antibiotics for life-threatening infection. The trouble is that the more obvious the benefit of the medical intervention, the less likely are doctors to ignore it. Thus, assessment of the
AUDIT
907
quality of medical care in deaths from massive post-partum haemorrhage usually hinges on whether blood transfusion was adequate or excessive, a somewhat more subtle and contentious point than failure to establish an intravenous line! There is, however, one area where clear guidelines can be laid down and where, in my view, audit has its greatest role. This is in the quality of administrative arrangements. Many of the appropriate processes are simply good manners but I believe these can be effectively audited. Thus audit can include such factors as 'were the general practitioner and midwife informed of a perinatal death before the patient's discharge?' or 'Have doctors recognized and acted upon an abnormal smear result?' It is my view that it is in these organizational matters that the health service most often falls down, and it is here that audit has the most to offer. In the above examples the accepted standard against which the process of care can be compared is very obvious. In other cases, however, the accepted standard should itself be audited to ensure that it is based on sound evidence. Chalmers (I. Chalmers, personal communication) pointed out the fundamental flaw in any form of audit that is restricted to an examination of the process of health care by citing an early example of medical audit. This involved censure of a Boston physician who had not prescribed the then standard treatment for pneumonia--blood letting. It is important that audit does not become an authoritarian tool to restrict the advance of medical knowledge. Thus, any requirement of particular policies for care or management within the health services should be based on good evidence that these policies are more likely to do good than harm. Evidence about the effects of care may take a variety of forms, including seeking the views of people who have used the health service. Here again caution is needed since we do not know how prior expectations among different groups of people influence their reactions to the health service. Other methods of care have been evaluated in properly controlled studies, thus generating evidence required for informed audit of practice. There is now good evidence that certain forms of care are more effective than others (for example, catgut sutures cause more pain following perineal wound repair, steroids promote fetal lung maturity and prophylactic antibiotics reduce the morbidity and financial costs of caesarean section, while oxytocic agents reduce the incidence of post-partum haemorrhage) (reviewed in Chalmers et al, 1989). However, many practices which are enshrined in obstetric dogma are of completely unproven value and may even be harmful; a good example is the glucose tolerance test. We therefore have good evidence on which to base certain policies and these can be used as a 'gold standard' against which the process of care can be judged. Other examples are much more tenuous and, before resorting to audit, more properly controlled, randomized experiments should be carried out. AUDIT OF AUDIT Audit itself needs to be audited. A good example of this is Barrett's caesarean
908
R.J.
LILFORD
section audit (Barrett et al, 1989) in which five auditors reviewed the case notes of a random sample of emergency caesarean sections for fetal distress. The initial audit showed a wide inter-observer variation, all five auditors agreeing in only 20% of cases. Furthermore, when re-presented with the same case notes, a marked inconsistency with first-time decisions was observed, the cumulative inconsistency rate being one case in four. This study reinforces the point that audit must be preceded by a validation of the audit process itself.
SUBTLE EFFECTS OF AUDIT There is one final value of audit which lies beyond the more straightforward benefits that may result from the direct educational and disciplining effects. This is a more nebulous but ironically possibly more important effect which results not from the information provided by audit, but from the very process of audit itself. This is the famous Hawthorne effect, whereby the very process of examining a human activity, improves that activity. Thus audit has a symbolic component to it: it sends a signal to health professionals that their work is being monitored and that they are accountable to their patients. In a rather more parochial British sense, audit forms an important component of the proposed health service reforms. The essence of these is the separation of responsibility for providing and purchasing medical services and the establishment of an internal market. However, many services, and antenatal care is an example, are semi-monopolies. The provider of these services might be in too strong a bargaining position for funds (cost per patient) if there was no link between remuneration and some attempt to measure the value of service in terms of outcome and consumer satisfaction.
ON-LINE AUDIT Lastly, audit is a retrospective process of quality control. It has educational and disciplining value and is designed to lower the chance that deficiencies or shortcomings will be repeated. The individual patient who may have had an increased risk of poor outcome as a result of the care standard which has been audited and found wanting will not benefit. It is for this reason that computerized history-taking and data transfer systems have great advantages in clinical care. They both educate and warn of pending risk in a particular patient prospectively rather than retrospectively. 1. 2. 3.
Hospital information systems reduce the risk of missing data (Chapter 3) by making laboratory or imaging data readily available. Pharmacy systems warn of drug interactions. Branching history systems ensure complete relevant data collection (Chapter 5).
AUDIT
4.
5. 6.
909
Low level artificial intelligence (action suggestions) remind the clinicians of appropriate tests and risks, given that humans do not have unlimited powers of recall and concentration and operate in an increasingly complex environment (Chapters 5, 10 and 12). More advanced artificial intelligence and Bayesian systems improve diagnostic precision in selected instances (Chapters 11 and 14). Electronic interfacing (Chapter 8) results in improved recognition of abnormal electrophysiological patterns (Chapter 9).
In all these cases the computer is used in the care of a particular patient, so that potential errors are covered in advance and patients are protected. Such systems also produce data for retrospective research and indeed they can be used to automate trials (Chapter 7) and thereby define better the appropriate standards of care for future on-line and retrospective audit. REFERENCES Barrett JFR, Jarvis GJ, Mcdonald HN, Buchan PC, Tyrrell S & Lilford RJ (1990) The inconsistencies o f clinical decisions revealed by an obstetric audit. Lancet (in press). Chalmers I, Enkin M & Keirse MJMC (1989) Effective care in pregnancy and childbirth: a synopsis for guiding practice and research. In Chalmers I, Enkin M & Keirse MJMC (eds) Effective Care in Pregnancy and Childbirth. Oxford: Oxford University Press.