- Email: [email protected]
Estimating with confidence the risk of rare adverse events, including those with observed rates of zero
Special Article
Estimating With Confidence the Risk of Rare Adverse Events, Including Those With Observed Rates of Zero Anthony M.-H. Ho, M.Sc., M.D., F.R.C.P.C., Peter W. Dion, Ph.D., M.D., F.R.C.P.C., Manoj K. Karmakar, M.D., F.R.C.A., and Anna Lee, Ph.D., M.P.H. Omission of a confidence interval (CI) associated with the risk of a serious complication can lead to inaccurate interpretation of risk data. The calculation of a CI for a risk or a single proportion typically uses the familiar Gaussian (normal) approximation. However, when the risk is small, “exact” methods or other special techniques should be used to avoid overshooting (risks that include values outside of [0,1]) and zero width interval degeneration. Computer programs and simple equations are available to construct CIs reasonably accurately. In the special case in which the complication has not occurred, the risk estimated with 95% confidence is no worse than 3/n, where n is the number of trials. Reg Anesth Pain Med 2002;27:207-210.
F
or any new treatment to become accepted into practice, its benefits should outweigh the risks. It is rare that a new therapy is totally without risks. When the risk in question is low, it may take a long time before an adverse event is reported. If the potential complication is serious, clinicians and patients are at a loss when trying to balance the potential benefits against the risk, as the latter is largely unknown. Ironically, such dilemma may deter the use of the new therapy, may hinder the planning and execution of clinical trials, and may frustrate patient recruitment, thus retarding the ac-
From the Department of Anaesthesia and Intensive Care (A.M.-H.H., M.K.K., A.L.), The Chinese University of Hong Kong, Shatin, NT, Hong Kong SAR, People’s Republic of China; and the Department of Anaesthesia (P.W.D.), St. Catharines General and Hotel Dieu Hospitals, St. Catharines, Ontario, Canada. Accepted for publication October 1, 2001. Supported by CUHK Direct Grant 2040785 and departmental funds. This work is attributable to the Chinese University of Hong Kong, Hong Kong, and the St. Catharines General and Hotel Dieu Hospitals, Canada. Reprint requests: Anthony M.-H. Ho, M.Sc., M.D., F.R.C.P.C., Department of Anaesthesia and Intensive Care, Prince of Wales Hospital, Shatin, NT, Hong Kong, People’s Republic of China. E-mail: [email protected] © 2002 by the American Society of Regional Anesthesia and Pain Medicine. 1098-7339/02/2702-0013$35.00/0 doi:10.1053/rapm.2002.30708
cumulation of data from which more accurate risk estimation can be derived. Quantification of the risk of a serious complication when it has not even occurred yet is useful because it provides objective information for making decisions. It serves to counterbalance the premature declaration of the safety of a new therapy based on zero occurrence of a plausible complication after only a limited number of observations. An example is that of the risks of common bile duct and vascular injuries in the early days of laparoscopic cholecystectomy and appendectomy, when enthusiastic surgeons were proclaiming the remarkable safety of the technique.1 On the other hand, it may counteract irrational condemnation of a new application because of a much feared complication. For example, Ho et al2 have estimated that the risk of spinal hematoma, a devastating, but as yet unreported complication from supplemental epidural analgesia for coronary artery bypass grafting involving cardiopulmonary bypass, may be comparable to the number of deaths from perioperative myocardial infarction that may be prevented by the technique.
Zero Occurrence By assuming that this plausible complication occurs randomly and independently, we will estimate its risk using a technique described by Hanley and Lippman-Hand.3 Let the risk of the complication occurring with each trial be r. The outcome of each
Regional Anesthesia and Pain Medicine, Vol 27, No 2 (March–April), 2002: pp 207–210
207
208
Regional Anesthesia and Pain Medicine Vol. 27 No. 2 March–April 2002 Table 1. Formulas for Calculating the Confidence Intervals of Proportions
Symbol
90% CI
95% CI
99% CI
A
2x ⫹ 2.71
2x ⫹ 3.84
2x ⫹ 6.64
B C
冑
冋 册
x 1.65 2.71⫹4x 1⫺ n 2(n ⫹ 2.71)
冑
冋 册
x 1.96 3.84⫹4x 1⫺ n 2(n ⫹ 3.84)
冑
冋 册
x 6.64⫹4x 1⫺ n 2(n ⫹ 6.64) 2.58
NOTE. The total number of observations is n, and the total number of the complication(s) of interest is x. The CI for the risk is given A⫺B A⫹B to . C C Abbreviation: CI, confidence interval.
by
trial of our treatment is either with or without this complication. Since the chance of each trial not resulting in the complication is (1 ⫺ r), the chance of n trials not resulting in any complication is (1 ⫺ r)n. Ideally, we would hope that the chance of getting no complication is unity, i.e., (1 ⫺ r)n ⫽ 1, meaning that r ⫽ 0. To believe that is to have been one of those who were certain that the common bile duct and major vessels would never be violated during laparoscopy, or to believe that a spinal injury is all but impossible after epidural instrumentation followed by profound anticoagulation, or that elevators never break down. Clearly, equating (1 ⫺ r)n to unity is reckless, and (1 ⫺ r)n must be less than one; in fact, to be cautious, much less. Since we do not want to underestimate r, let us then ask how large can r be, and still be consistent with the experience of n uncomplicated trials. If the r we pick gives a chance of one in a million of having n uncomplicated trials [(1 ⫺ r)n ⫽ 0.000001], it would be rather close to 1 (depending, of course, on the magnitude of n). This is overly cautious. The true value for r must be smaller, if for no other reason than the fact that we have had n good trials already. So let us pick a smaller r to correspond to the probability of n uncomplicated trials of, say, 1 in 20, i.e., (1 ⫺ r)n ⫽ 0.05. In other words, to obtain a conservative, yet realistic, estimate of r, we assume that the chance of no complication after the next n trials is 0.05. The value for r derived from this equation would correspond to the maximum estimate with 95% confidence. The solution to this equation is shown in Appendix A. For n ⬎ 30, it can be shown2,3 that r⬇
3 . n
Rare Occurrence and the Importance of the Confidence Interval So far, we have estimated the risk of plausible adverse events with zero occurrence. For 100 uncomplicated trials, this risk is 0.03. What if among the 100 trials, the complication in question occurred twice? Most people would have concluded
that the risk is 0.02, which is less than the risk derived from n trials with no complication at all! The way out of this paradox is to recognize that 0.02 is the mean risk estimate, whereas the 0.03 value calculated earlier is the worst estimate with 95% confidence. Reporting only the mean risk or proportion of observations with the complication, as is often the case, is incomplete. To obtain a proper estimate of the risk based on x complications after n trials, taking into consideration the imprecision caused by random variation in the sample, we must determine a confidence interval (CI) around that proportion. Note that in the zero occurrence calculation above, the estimate r⬇
3 n
is the upper limit of a 1-sided 95% CI, which is equivalent to the upper limit of a 2-sided 90% CI. Typically, a CI can be constructed by taking a multiple of the standard error on either side of the mean or sample proportion x . n The problem when x is small or even zero is that the CI includes negative values (clearly impossible when the smallest value for x is zero). Alternative techniques are available to calculate the CI with high degrees of accuracy while avoiding this pitfall.4 Examples of these techniques include Clopper and Pearson’s highly conservative binomial-based “exact” method,5 Blyth and Still’s asymptotic method with continuity correction,6 etc. Many different statistical programs (e.g., StatXact [Cytel Software Corp, Cambridge, MA], Confidence Interval Analysis [a software package that accompanies the book by Altman et al7]) utilizing some of these techniques are now available. The mathematics behind these techniques can be quite involved, and the best one to use for any set of data is beyond the scope of this discussion. For those who do not have access to such statistical packages, Newcombe and Altman7 have provided the equations that we have further simplified and presented in Table 1.
Estimating Confidence Intervals
These computer programs and equations can be used to estimate the CI of a risk or proportion, irrespective of whether it is high or low, including the special situation when there has been zero complication. We have nonetheless devoted a separate section on zero occurrence because while most people have no trouble appreciating the mean risk of a complication of x , n many have difficulties understanding what to do when x is zero. CIs are typically expressed as 90%, 95%, or 99%. The 95% confidence level for zero occurrence event calculated earlier represents the upper limit of a 90% CI. In keeping with the correct classical (frequentist) statistical interpretation of CI, we point out that a 95% CI does not mean that the true risk lies within the CI, it means that if the true risk lies outside the 95% CI, the likelihood of obtaining the data observed is less than or equal to 5%. Some Pitfalls Associated With Not Emphasizing the CIs of Risk Estimates The lack of reporting of CIs in medical literature often leads to propagation of misleading data. For example, an important complication has occurred once after 50 trials in 1 study, and has occurred twice after 50 trials in another. Since the authors of both articles have not reported the associated CIs, subsequent authors may simply quote the risk of this complication as 2% to 4%. Even though the combined data yield a 95% CI of 1.0% to 8.2%, it is the 2% to 4% figure that may get quoted in research, review papers, and textbook chapters. Busy clinicians reading the reviews may simply remember 2% to 4% as if the numbers 2 and 4 represented the lower and upper borders of some CI, typically 95%. Stating the CI is important because it reflects the sample size on which the risk calculation is based. For example, the 95% CI for 1 complication after 100 trials is 0.18% to 5.45% (range, 5.3%), whereas the 95% CI for 10 complications after 1,000 trials is 0.54% to 1.83% (range, 1.3%). The means are the same, but the estimated range of the risk is much smaller (by a factor of 4) when the sample size is increased by a factor of 10. Interestingly, this effect of the sample size on CI is less pronounced for nonrare complications, underscoring the importance of increased data collection for all complications, particularly the rare ones. For comparison, the 95% CI for 25 complications after 100 trials is 17.6% to 34.3% (range, 16.7%), and the 95% CI for 250 complications after 1,000 trials is 22.4% to 27.8% (range, 5.4%, an improvement by a factor of 3 instead of 4 as illustrated above for a rarer complication).
•
Ho et al.
209
Often, there is more than one technique to achieve the same therapeutic goal, and the choice may partly depend on which one has the lowest risk. Studies directly comparing one technique with another are not always available, and clinicians may have to extract the data from different reports. Sometimes, technique A may appear safer than technique B because it has a lower mean risk, but after constructing 95% CIs around those proportions, one may find that the risks are actually similar because a substantial part of their 95% CIs overlap. Likewise, 2 different techniques with similar means may have different maximum risk estimates. Unfortunately, authors often do not report the CIs, while other authors propagate the incomplete information by quoting them. Application to Regional Anesthesia and Pain Medicine The subject of thoracic epidural anesthesia/analgesia (TEA) in cardiac surgery involving cardiopulmonary bypass is controversial. The concern over spinal epidural hematoma is so overwhelming that many anesthesiologists simply would not consider that option, even as an increasing number of centers throughout the world are using the technique.8 What is the risk of this dreaded complication? As of 1998, there might have been some 5,000 cases of TEA performed in patients undergoing cardiac surgery.2 The trend has accelerated since and today the total number could easily have doubled. No case of spinal hematoma has ever been reported. Based on the concept presented earlier on “zero occurrence,” the maximum risk with 95% and 99% confidence of spinal epidural hematoma is approximately 1:3,300 and 1:2,200, respectively. These numbers are possibly comparable to the numbers of deaths from postoperative myocardial infarction (MI) that could be prevented by the use of TEA in coronary artery bypass grafting (assuming the incidence of postoperative MI of 4% can be reduced to 3.5%, and an MI mortality rate of 13%).2 For anesthesia and pain management of certain thoracoabdominal surgeries and pathologies (e.g., multiple fractured ribs), there are several regional techniques that could be used, including intercostal,9 interpleural,10 and paravertebral blocks.11,12 Table 2 shows the associated incidences of symptomatic pneumothorax. Note that some of the 95% CIs overlap, suggesting that, in spite of the noticeable differences in the means, the risk of pneumothorax is, based on conventional statistical wisdom, similar between thoracic paravertebral and interpleural blocks, and similar between intercostal and paravertebral blocks. It is reasonable to expect that as more data become available, significant differences may emerge.
210
Regional Anesthesia and Pain Medicine Vol. 27 No. 2 March–April 2002 Table 2. Incidence of Symptomatic Pneumothorax Based on Reported Series
Regional Technique Intercostal Interpleural Paravertebral
Risk (%)
Source of Data
No. of Cases
No. of Pneumothorax
Mean
90% CI
95% CI
99% CI
Moore et al.9 Strømskag et al.10 Lo¨nnqvist et al.,11 Coveney et al.12
13,525 703 523
12 14 2
0.09 2 0.38
.06-.14 1.3-3.1 .13-1.2
.05-.16 1.2-3.3 .1-1.4
.04-.18 1.0-3.9 .07-1.9
NOTE. The chance of pneumothorax is very much operator dependent, and different investigators may have different thresholds/criteria for looking for pneumothorax. Theoretically, all interpleural blocks result in pneumothorax. Abbreviation: CI, confidence interval.
Closing Remark and Summary The very nature of anesthesiology is risky. We frequently subject patients to a variety of pharmacological, physiological, and invasive neural challenges, most of the time to allow surgery, a further invasive technique, to take place.13 Some of the potential complications and side effects are very serious, and it is important to be able to quote not only the means of their occurrences, but also the CIs around them. Based on a P value of .05, we routinely accept null hypotheses and reject alternative hypotheses, and vice versa, with major implications. There is no reason why, when it comes to important clinical decisions involving interventions with potentially serious complications, we should not apply the same principle and consider the CIs of the plausible risks. As our specialty evolves, new treatment modalities and techniques are constantly being developed. The rational utilization of these innovations depends, in part, on our ability to quantify the rates of potential complications, even when they are rare or have had zero occurrence. This process could be greatly facilitated by vastly expanding existing and establishing new databases on some of the treatments we have been or will be administering, and call on all clinicians to contribute information on the number of trials and the frequencies of complications. Appendix A. Solving (1 ⫺ r)n ⫽ 0.05 Rearranging the above 2 : r ⫽ 1 ⫺ eln(y) ⫽⫺
ln共0.05兲 [ln(0.05)] 2 ⫺ n 2n 2
⫺
关ln共0.05兲兴 3 [ln(0.05)] 4 ⫺ ⫺... 3 6n 24n 4
⬇
3 27 81 9 ⫹ ⫹ ⫹... ⫹ n 2n 2 6n 3 24n 4
For n ⬎ 30, the first term above is much larger than the rest and r ⫽ 3/n. For n ⬍ 20-30, one would have to plug the value of n into the above equation. For example, if one has had zero occurrence of a complication in question after 10 trials, the maximum risk of
that complication estimated with 95% confidence is 0.3 ⫹ 0.045 ⫹ 0.0045 ⫹ 0.0003375 ⫹ . . . ⬃ 0.35.
References 1. Eypasch E, Lefering R, Kum CK, Troidl H. Probability of adverse events that have not yet occurred: A statistical reminder. BMJ 1995;311:619-620. 2. Ho AMH, Chung DC, Joynt GM. Neuraxial blockade and hematoma in cardiac surgery: Estimating the risk of a rare adverse event that has not (yet) occurred. Chest 2000;117:551-555. 3. Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA 1983;249:1743-1745. 4. Newcombe RG. Two-sided confidence intervals for the single proportion: Comparison of seven methods. Stat Med 1998;17:857-872. 5. Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 1934;26:404-413. 6. Blyth CR, Still HA. Binomial confidence intervals. J Am Stat Assoc 1983;78:108-116. 7. Newcombe RG, Altman DG. Proportions and their differences. In: Altman DG, Machin D, Bryant TN, Gardner MJ, eds. Statistics with Confidence, 2nd ed. Bristol, England: BMJ; 2000:45-56. 8. Goldstein S, Dean D, Kim SJ, Cocozello K, Grofsik J, Silver P, Cody RP. A survey of spinal and epidural techniques in adult cardiac surgery. J Cardiothorac Vasc Anesth 2001;15:158-168. 9. Moore DC, Bush WH, Scurlock JE. Intercostal nerve block: A roentgenographic anatomic study of technique and absorption in humans. Anesth Analg 1980; 59:815-825. 10. Strømskag KE, Minor B, Steen PA. Side effects and complications of interpleural analgesia: An update. Acta Anesthesiol Scand 1990;34:473-477. 11. Lo¨nnqvist PA, MacKenzie J, Soni AK. Paravertebral blockade: Failure rate and complications. Anaesthesia 1995;50:813-815. 12. Coveney E, Weltz CR, Greengrass R, Iglehart JD, Leight GS, Steele SM, Lyerly HK. Use of paravertebral block anesthesia in the surgical management of breast cancer: Experience in 156 cases. Ann Surg 1998;227:496-501. 13. Mason RA. The case report—an endangered species? Anaesthesia 2001;56:99-102.
Estimating With Confidence the Risk of Rare Adverse Events, Including Those With Observed Rates of Zero Anthony M.-H. Ho, M.Sc., M.D., F.R.C.P.C., Peter W. Dion, Ph.D., M.D., F.R.C.P.C., Manoj K. Karmakar, M.D., F.R.C.A., and Anna Lee, Ph.D., M.P.H. Omission of a confidence interval (CI) associated with the risk of a serious complication can lead to inaccurate interpretation of risk data. The calculation of a CI for a risk or a single proportion typically uses the familiar Gaussian (normal) approximation. However, when the risk is small, “exact” methods or other special techniques should be used to avoid overshooting (risks that include values outside of [0,1]) and zero width interval degeneration. Computer programs and simple equations are available to construct CIs reasonably accurately. In the special case in which the complication has not occurred, the risk estimated with 95% confidence is no worse than 3/n, where n is the number of trials. Reg Anesth Pain Med 2002;27:207-210.
F
or any new treatment to become accepted into practice, its benefits should outweigh the risks. It is rare that a new therapy is totally without risks. When the risk in question is low, it may take a long time before an adverse event is reported. If the potential complication is serious, clinicians and patients are at a loss when trying to balance the potential benefits against the risk, as the latter is largely unknown. Ironically, such dilemma may deter the use of the new therapy, may hinder the planning and execution of clinical trials, and may frustrate patient recruitment, thus retarding the ac-
From the Department of Anaesthesia and Intensive Care (A.M.-H.H., M.K.K., A.L.), The Chinese University of Hong Kong, Shatin, NT, Hong Kong SAR, People’s Republic of China; and the Department of Anaesthesia (P.W.D.), St. Catharines General and Hotel Dieu Hospitals, St. Catharines, Ontario, Canada. Accepted for publication October 1, 2001. Supported by CUHK Direct Grant 2040785 and departmental funds. This work is attributable to the Chinese University of Hong Kong, Hong Kong, and the St. Catharines General and Hotel Dieu Hospitals, Canada. Reprint requests: Anthony M.-H. Ho, M.Sc., M.D., F.R.C.P.C., Department of Anaesthesia and Intensive Care, Prince of Wales Hospital, Shatin, NT, Hong Kong, People’s Republic of China. E-mail: [email protected] © 2002 by the American Society of Regional Anesthesia and Pain Medicine. 1098-7339/02/2702-0013$35.00/0 doi:10.1053/rapm.2002.30708
cumulation of data from which more accurate risk estimation can be derived. Quantification of the risk of a serious complication when it has not even occurred yet is useful because it provides objective information for making decisions. It serves to counterbalance the premature declaration of the safety of a new therapy based on zero occurrence of a plausible complication after only a limited number of observations. An example is that of the risks of common bile duct and vascular injuries in the early days of laparoscopic cholecystectomy and appendectomy, when enthusiastic surgeons were proclaiming the remarkable safety of the technique.1 On the other hand, it may counteract irrational condemnation of a new application because of a much feared complication. For example, Ho et al2 have estimated that the risk of spinal hematoma, a devastating, but as yet unreported complication from supplemental epidural analgesia for coronary artery bypass grafting involving cardiopulmonary bypass, may be comparable to the number of deaths from perioperative myocardial infarction that may be prevented by the technique.
Zero Occurrence By assuming that this plausible complication occurs randomly and independently, we will estimate its risk using a technique described by Hanley and Lippman-Hand.3 Let the risk of the complication occurring with each trial be r. The outcome of each
Regional Anesthesia and Pain Medicine, Vol 27, No 2 (March–April), 2002: pp 207–210
207
208
Regional Anesthesia and Pain Medicine Vol. 27 No. 2 March–April 2002 Table 1. Formulas for Calculating the Confidence Intervals of Proportions
Symbol
90% CI
95% CI
99% CI
A
2x ⫹ 2.71
2x ⫹ 3.84
2x ⫹ 6.64
B C
冑
冋 册
x 1.65 2.71⫹4x 1⫺ n 2(n ⫹ 2.71)
冑
冋 册
x 1.96 3.84⫹4x 1⫺ n 2(n ⫹ 3.84)
冑
冋 册
x 6.64⫹4x 1⫺ n 2(n ⫹ 6.64) 2.58
NOTE. The total number of observations is n, and the total number of the complication(s) of interest is x. The CI for the risk is given A⫺B A⫹B to . C C Abbreviation: CI, confidence interval.
by
trial of our treatment is either with or without this complication. Since the chance of each trial not resulting in the complication is (1 ⫺ r), the chance of n trials not resulting in any complication is (1 ⫺ r)n. Ideally, we would hope that the chance of getting no complication is unity, i.e., (1 ⫺ r)n ⫽ 1, meaning that r ⫽ 0. To believe that is to have been one of those who were certain that the common bile duct and major vessels would never be violated during laparoscopy, or to believe that a spinal injury is all but impossible after epidural instrumentation followed by profound anticoagulation, or that elevators never break down. Clearly, equating (1 ⫺ r)n to unity is reckless, and (1 ⫺ r)n must be less than one; in fact, to be cautious, much less. Since we do not want to underestimate r, let us then ask how large can r be, and still be consistent with the experience of n uncomplicated trials. If the r we pick gives a chance of one in a million of having n uncomplicated trials [(1 ⫺ r)n ⫽ 0.000001], it would be rather close to 1 (depending, of course, on the magnitude of n). This is overly cautious. The true value for r must be smaller, if for no other reason than the fact that we have had n good trials already. So let us pick a smaller r to correspond to the probability of n uncomplicated trials of, say, 1 in 20, i.e., (1 ⫺ r)n ⫽ 0.05. In other words, to obtain a conservative, yet realistic, estimate of r, we assume that the chance of no complication after the next n trials is 0.05. The value for r derived from this equation would correspond to the maximum estimate with 95% confidence. The solution to this equation is shown in Appendix A. For n ⬎ 30, it can be shown2,3 that r⬇
3 . n
Rare Occurrence and the Importance of the Confidence Interval So far, we have estimated the risk of plausible adverse events with zero occurrence. For 100 uncomplicated trials, this risk is 0.03. What if among the 100 trials, the complication in question occurred twice? Most people would have concluded
that the risk is 0.02, which is less than the risk derived from n trials with no complication at all! The way out of this paradox is to recognize that 0.02 is the mean risk estimate, whereas the 0.03 value calculated earlier is the worst estimate with 95% confidence. Reporting only the mean risk or proportion of observations with the complication, as is often the case, is incomplete. To obtain a proper estimate of the risk based on x complications after n trials, taking into consideration the imprecision caused by random variation in the sample, we must determine a confidence interval (CI) around that proportion. Note that in the zero occurrence calculation above, the estimate r⬇
3 n
is the upper limit of a 1-sided 95% CI, which is equivalent to the upper limit of a 2-sided 90% CI. Typically, a CI can be constructed by taking a multiple of the standard error on either side of the mean or sample proportion x . n The problem when x is small or even zero is that the CI includes negative values (clearly impossible when the smallest value for x is zero). Alternative techniques are available to calculate the CI with high degrees of accuracy while avoiding this pitfall.4 Examples of these techniques include Clopper and Pearson’s highly conservative binomial-based “exact” method,5 Blyth and Still’s asymptotic method with continuity correction,6 etc. Many different statistical programs (e.g., StatXact [Cytel Software Corp, Cambridge, MA], Confidence Interval Analysis [a software package that accompanies the book by Altman et al7]) utilizing some of these techniques are now available. The mathematics behind these techniques can be quite involved, and the best one to use for any set of data is beyond the scope of this discussion. For those who do not have access to such statistical packages, Newcombe and Altman7 have provided the equations that we have further simplified and presented in Table 1.
Estimating Confidence Intervals
These computer programs and equations can be used to estimate the CI of a risk or proportion, irrespective of whether it is high or low, including the special situation when there has been zero complication. We have nonetheless devoted a separate section on zero occurrence because while most people have no trouble appreciating the mean risk of a complication of x , n many have difficulties understanding what to do when x is zero. CIs are typically expressed as 90%, 95%, or 99%. The 95% confidence level for zero occurrence event calculated earlier represents the upper limit of a 90% CI. In keeping with the correct classical (frequentist) statistical interpretation of CI, we point out that a 95% CI does not mean that the true risk lies within the CI, it means that if the true risk lies outside the 95% CI, the likelihood of obtaining the data observed is less than or equal to 5%. Some Pitfalls Associated With Not Emphasizing the CIs of Risk Estimates The lack of reporting of CIs in medical literature often leads to propagation of misleading data. For example, an important complication has occurred once after 50 trials in 1 study, and has occurred twice after 50 trials in another. Since the authors of both articles have not reported the associated CIs, subsequent authors may simply quote the risk of this complication as 2% to 4%. Even though the combined data yield a 95% CI of 1.0% to 8.2%, it is the 2% to 4% figure that may get quoted in research, review papers, and textbook chapters. Busy clinicians reading the reviews may simply remember 2% to 4% as if the numbers 2 and 4 represented the lower and upper borders of some CI, typically 95%. Stating the CI is important because it reflects the sample size on which the risk calculation is based. For example, the 95% CI for 1 complication after 100 trials is 0.18% to 5.45% (range, 5.3%), whereas the 95% CI for 10 complications after 1,000 trials is 0.54% to 1.83% (range, 1.3%). The means are the same, but the estimated range of the risk is much smaller (by a factor of 4) when the sample size is increased by a factor of 10. Interestingly, this effect of the sample size on CI is less pronounced for nonrare complications, underscoring the importance of increased data collection for all complications, particularly the rare ones. For comparison, the 95% CI for 25 complications after 100 trials is 17.6% to 34.3% (range, 16.7%), and the 95% CI for 250 complications after 1,000 trials is 22.4% to 27.8% (range, 5.4%, an improvement by a factor of 3 instead of 4 as illustrated above for a rarer complication).
•
Ho et al.
209
Often, there is more than one technique to achieve the same therapeutic goal, and the choice may partly depend on which one has the lowest risk. Studies directly comparing one technique with another are not always available, and clinicians may have to extract the data from different reports. Sometimes, technique A may appear safer than technique B because it has a lower mean risk, but after constructing 95% CIs around those proportions, one may find that the risks are actually similar because a substantial part of their 95% CIs overlap. Likewise, 2 different techniques with similar means may have different maximum risk estimates. Unfortunately, authors often do not report the CIs, while other authors propagate the incomplete information by quoting them. Application to Regional Anesthesia and Pain Medicine The subject of thoracic epidural anesthesia/analgesia (TEA) in cardiac surgery involving cardiopulmonary bypass is controversial. The concern over spinal epidural hematoma is so overwhelming that many anesthesiologists simply would not consider that option, even as an increasing number of centers throughout the world are using the technique.8 What is the risk of this dreaded complication? As of 1998, there might have been some 5,000 cases of TEA performed in patients undergoing cardiac surgery.2 The trend has accelerated since and today the total number could easily have doubled. No case of spinal hematoma has ever been reported. Based on the concept presented earlier on “zero occurrence,” the maximum risk with 95% and 99% confidence of spinal epidural hematoma is approximately 1:3,300 and 1:2,200, respectively. These numbers are possibly comparable to the numbers of deaths from postoperative myocardial infarction (MI) that could be prevented by the use of TEA in coronary artery bypass grafting (assuming the incidence of postoperative MI of 4% can be reduced to 3.5%, and an MI mortality rate of 13%).2 For anesthesia and pain management of certain thoracoabdominal surgeries and pathologies (e.g., multiple fractured ribs), there are several regional techniques that could be used, including intercostal,9 interpleural,10 and paravertebral blocks.11,12 Table 2 shows the associated incidences of symptomatic pneumothorax. Note that some of the 95% CIs overlap, suggesting that, in spite of the noticeable differences in the means, the risk of pneumothorax is, based on conventional statistical wisdom, similar between thoracic paravertebral and interpleural blocks, and similar between intercostal and paravertebral blocks. It is reasonable to expect that as more data become available, significant differences may emerge.
210
Regional Anesthesia and Pain Medicine Vol. 27 No. 2 March–April 2002 Table 2. Incidence of Symptomatic Pneumothorax Based on Reported Series
Regional Technique Intercostal Interpleural Paravertebral
Risk (%)
Source of Data
No. of Cases
No. of Pneumothorax
Mean
90% CI
95% CI
99% CI
Moore et al.9 Strømskag et al.10 Lo¨nnqvist et al.,11 Coveney et al.12
13,525 703 523
12 14 2
0.09 2 0.38
.06-.14 1.3-3.1 .13-1.2
.05-.16 1.2-3.3 .1-1.4
.04-.18 1.0-3.9 .07-1.9
NOTE. The chance of pneumothorax is very much operator dependent, and different investigators may have different thresholds/criteria for looking for pneumothorax. Theoretically, all interpleural blocks result in pneumothorax. Abbreviation: CI, confidence interval.
Closing Remark and Summary The very nature of anesthesiology is risky. We frequently subject patients to a variety of pharmacological, physiological, and invasive neural challenges, most of the time to allow surgery, a further invasive technique, to take place.13 Some of the potential complications and side effects are very serious, and it is important to be able to quote not only the means of their occurrences, but also the CIs around them. Based on a P value of .05, we routinely accept null hypotheses and reject alternative hypotheses, and vice versa, with major implications. There is no reason why, when it comes to important clinical decisions involving interventions with potentially serious complications, we should not apply the same principle and consider the CIs of the plausible risks. As our specialty evolves, new treatment modalities and techniques are constantly being developed. The rational utilization of these innovations depends, in part, on our ability to quantify the rates of potential complications, even when they are rare or have had zero occurrence. This process could be greatly facilitated by vastly expanding existing and establishing new databases on some of the treatments we have been or will be administering, and call on all clinicians to contribute information on the number of trials and the frequencies of complications. Appendix A. Solving (1 ⫺ r)n ⫽ 0.05 Rearranging the above 2 : r ⫽ 1 ⫺ eln(y) ⫽⫺
ln共0.05兲 [ln(0.05)] 2 ⫺ n 2n 2
⫺
关ln共0.05兲兴 3 [ln(0.05)] 4 ⫺ ⫺... 3 6n 24n 4
⬇
3 27 81 9 ⫹ ⫹ ⫹... ⫹ n 2n 2 6n 3 24n 4
For n ⬎ 30, the first term above is much larger than the rest and r ⫽ 3/n. For n ⬍ 20-30, one would have to plug the value of n into the above equation. For example, if one has had zero occurrence of a complication in question after 10 trials, the maximum risk of
that complication estimated with 95% confidence is 0.3 ⫹ 0.045 ⫹ 0.0045 ⫹ 0.0003375 ⫹ . . . ⬃ 0.35.
References 1. Eypasch E, Lefering R, Kum CK, Troidl H. Probability of adverse events that have not yet occurred: A statistical reminder. BMJ 1995;311:619-620. 2. Ho AMH, Chung DC, Joynt GM. Neuraxial blockade and hematoma in cardiac surgery: Estimating the risk of a rare adverse event that has not (yet) occurred. Chest 2000;117:551-555. 3. Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA 1983;249:1743-1745. 4. Newcombe RG. Two-sided confidence intervals for the single proportion: Comparison of seven methods. Stat Med 1998;17:857-872. 5. Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 1934;26:404-413. 6. Blyth CR, Still HA. Binomial confidence intervals. J Am Stat Assoc 1983;78:108-116. 7. Newcombe RG, Altman DG. Proportions and their differences. In: Altman DG, Machin D, Bryant TN, Gardner MJ, eds. Statistics with Confidence, 2nd ed. Bristol, England: BMJ; 2000:45-56. 8. Goldstein S, Dean D, Kim SJ, Cocozello K, Grofsik J, Silver P, Cody RP. A survey of spinal and epidural techniques in adult cardiac surgery. J Cardiothorac Vasc Anesth 2001;15:158-168. 9. Moore DC, Bush WH, Scurlock JE. Intercostal nerve block: A roentgenographic anatomic study of technique and absorption in humans. Anesth Analg 1980; 59:815-825. 10. Strømskag KE, Minor B, Steen PA. Side effects and complications of interpleural analgesia: An update. Acta Anesthesiol Scand 1990;34:473-477. 11. Lo¨nnqvist PA, MacKenzie J, Soni AK. Paravertebral blockade: Failure rate and complications. Anaesthesia 1995;50:813-815. 12. Coveney E, Weltz CR, Greengrass R, Iglehart JD, Leight GS, Steele SM, Lyerly HK. Use of paravertebral block anesthesia in the surgical management of breast cancer: Experience in 156 cases. Ann Surg 1998;227:496-501. 13. Mason RA. The case report—an endangered species? Anaesthesia 2001;56:99-102.