- Email: [email protected]
Embryology revisited
COMMENTARY
previously proposed by Baker et al,7 and expands and improves on statistical methods for monitoring such events.8,9 Whilst it may be problematic from a political and ethical standpoint to prospectively monitor health outcomes of populations of patients associated with individual physicians and practices, Aylin and colleagues provide an innovative step forward in terms of developing these statistical methods. As noted by Aylin, there are both costs and benefits associated with designing such systems. The largest cost associated with Aylin’s proposed system is the need to investigate a large number of false positives (innocent physicians) before the system will identify a true positive (guilty physician).7 This cost has important political ramifications and may undermine the good will of many physicians who will eventually be identified as false positives, but will need to be investigated before they are cleared of the initial suspicion. In addition to this cost, there are well-demonstrated short-term and long-term costs and benefits associated with safety monitoring. As these types of programmes evolve, the cost-benefit ratios will change.5 For example, in adverse-event monitoring of vaccines, there is often an initial decrease in public confidence when previously unidentified vaccine-safety events are detected in surveillance systems. On the other hand, in the long term, there are beneficial increases in public confidence because the public is aware and reassured by the fact that publichealth officials are actively monitoring and reporting these events.10 Aylin and colleagues address an important issue about how to appropriately monitor health outcomes when there are missing factors that would explain a large proportion of the variation in health outcomes. Often these missing factors are associated with the health status of the population of patients. By not making adjustments to physicians’ baselines and accounting for deviations that are associated with case-mix, the monitoring systems would unnecessarily implicate many innocent physicians. Statistically there are several important issues that need to be considered for these types of surveillance systems. Determining successful detection and false-alarm rates that are acceptable is extremely difficult and is dependent on the case-mix of individual physicians and the category of death being investigated. Aylin and colleagues set the successful detection rate at 97% and the false-alarm rate at 5%. Successful detection and false-alarm rates are influenced by several factors, including the base rate, sensitivity, and specificity of the health outcome. These detection rates can be adjusted or modified in several ways, including use of alternative thresholds for alarms, adjusting for case-mix and other covariates, or adjusting for overdispersion (extra variation) in the health outcome due to unmeasured factors. Aylin and colleagues propose a method that adjusts for overdispersion and use the median estimate of overdispersion from a 7-year baseline period. We assume there are true “out-of-control” processes (guilty physicians) included in the baseline period, and as a result, the estimates would likely be biased, resulting in conservative statistical tests, higher thresholds of alarm, and fewer false positives. It may be possible to adjust for these biases by eliminating the guilty physicians after they have been identified retrospectively. Data from the remaining, primarily innocent physicians would allow for appropriate estimation of “in-control” processes, and thence estimation of unbiased thresholds. In general, the method proposed by Aylin and colleagues, which adjusts for overdispersion, is a 418
reasonable and practical way to compensate for the inability to adequately adjust for risk. When possible, every effort should be made to collect data necessary for the appropriate risk-adjusted analysis. Systems designed to detect aberrations assist in reducing the time and effort needed to monitor and audit publichealth data for the investigation of unusual events, whether they are unusual increases in deaths, hospitalisations, or respiratory events. These systems can be phased in with conservative thresholds that limit the number of false positives and highlight the potential benefits of these systems. In the long run, these systems increase public confidence by reducing the costs associated with auditing health outcomes through the use of efficient methods and timely surveillance systems. We have no conflicts of interest to declare.
*William Thompson, Lori Hutwagner, Margarette Kolczak National Immunization Program, US Centers for Disease Control and Prevention, Atlanta, GA 30333, USA (e-mail: [email protected]) 1
Farr W. In: Vital statistics, 1885, London: Office of the Sanitary Institute, 1948: 330–33. 2 Farrington CP, Andrews NJ, Beale AD, Catchpole MA. A statistical algorithm for the early detection of outbreaks of infectious disease. J R Stat Soc A 1996; 159: 547–63. 3 Stroup DF, Williamson GD, Herndon JL, Karon J. Detection of aberrations in the occurrence of notifiable diseases surveillance data. Stat Med 1989; 8: 323–29. 4 Evans S. Statistical methods of signal detection. In: Mann RD, Andrews EB, eds. Pharmacovigilance. Chichester: Wiley, 2002: 273–79. 5 Chen R. Vaccine risks, perceived and unknown. Vaccine 1999; 17: S41–S46. 6 Hutwagner L, Thompson WW, Seeman GM, Treadwell T. The bioterrorism preparedness and response early aberration reporting system (EARS). J Urban Health 2003; S1: i89–i96. 7 Baker R, Jones DR, Goldblatt P. Monitoring mortality rates in general practice after Shipman. BMJ 2003; 326: 274–76. 8 Steiner SH, Cook RJ, Farewell VT, Treasure T. Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics 2000; 1: 441–52. 9 Grigg OA, Farewell VT, Spiegelhalter DJ. Use of risk-adjusted CUSUM and RSPRT charts for monitoring in medical contexts. Stat Meth Med Res 2003; 12: 147–70. 10 McPhillips HA, Davis RL, Marcuse EK, Taylor JA. The rotavirus vaccine's withdrawal and physicians’ trust in vaccine safety mechanisms. Arch Pediatr Adolesc Med 2001; 155: 1051–56.
Embryology revisited See page 477 Most doctors have a dim memory of embryology teaching at medical school. In a four-paper series starting in this week’s Lancet, we have tried to highlight some of the more fascinating findings in what now goes under the rather more trendy name of developmental biology. The first paper by Dian Donnai and Andrew Read discusses the role of the clinician in pinpointing some of the syndromes that have helped to elucidate developmental pathways. The next paper by William Horton describes ways of making animal models that help in understanding complex human syndromes, particularly those that affect the skeleton, and in the next paper Frances Goodman details the pathways involved in body patterning and how comparisons between species have been so revealing. The final paper, by our adviser on the series, Judith Hall, discusses the complex events involved in twinning, and brings the topic right back to the clinic. Virginia Barbour The Lancet, London NW1 7BY, UK
THE LANCET • Vol 362 • August 9, 2003 • www.thelancet.com
For personal use. Only reproduce with permission from The Lancet
previously proposed by Baker et al,7 and expands and improves on statistical methods for monitoring such events.8,9 Whilst it may be problematic from a political and ethical standpoint to prospectively monitor health outcomes of populations of patients associated with individual physicians and practices, Aylin and colleagues provide an innovative step forward in terms of developing these statistical methods. As noted by Aylin, there are both costs and benefits associated with designing such systems. The largest cost associated with Aylin’s proposed system is the need to investigate a large number of false positives (innocent physicians) before the system will identify a true positive (guilty physician).7 This cost has important political ramifications and may undermine the good will of many physicians who will eventually be identified as false positives, but will need to be investigated before they are cleared of the initial suspicion. In addition to this cost, there are well-demonstrated short-term and long-term costs and benefits associated with safety monitoring. As these types of programmes evolve, the cost-benefit ratios will change.5 For example, in adverse-event monitoring of vaccines, there is often an initial decrease in public confidence when previously unidentified vaccine-safety events are detected in surveillance systems. On the other hand, in the long term, there are beneficial increases in public confidence because the public is aware and reassured by the fact that publichealth officials are actively monitoring and reporting these events.10 Aylin and colleagues address an important issue about how to appropriately monitor health outcomes when there are missing factors that would explain a large proportion of the variation in health outcomes. Often these missing factors are associated with the health status of the population of patients. By not making adjustments to physicians’ baselines and accounting for deviations that are associated with case-mix, the monitoring systems would unnecessarily implicate many innocent physicians. Statistically there are several important issues that need to be considered for these types of surveillance systems. Determining successful detection and false-alarm rates that are acceptable is extremely difficult and is dependent on the case-mix of individual physicians and the category of death being investigated. Aylin and colleagues set the successful detection rate at 97% and the false-alarm rate at 5%. Successful detection and false-alarm rates are influenced by several factors, including the base rate, sensitivity, and specificity of the health outcome. These detection rates can be adjusted or modified in several ways, including use of alternative thresholds for alarms, adjusting for case-mix and other covariates, or adjusting for overdispersion (extra variation) in the health outcome due to unmeasured factors. Aylin and colleagues propose a method that adjusts for overdispersion and use the median estimate of overdispersion from a 7-year baseline period. We assume there are true “out-of-control” processes (guilty physicians) included in the baseline period, and as a result, the estimates would likely be biased, resulting in conservative statistical tests, higher thresholds of alarm, and fewer false positives. It may be possible to adjust for these biases by eliminating the guilty physicians after they have been identified retrospectively. Data from the remaining, primarily innocent physicians would allow for appropriate estimation of “in-control” processes, and thence estimation of unbiased thresholds. In general, the method proposed by Aylin and colleagues, which adjusts for overdispersion, is a 418
reasonable and practical way to compensate for the inability to adequately adjust for risk. When possible, every effort should be made to collect data necessary for the appropriate risk-adjusted analysis. Systems designed to detect aberrations assist in reducing the time and effort needed to monitor and audit publichealth data for the investigation of unusual events, whether they are unusual increases in deaths, hospitalisations, or respiratory events. These systems can be phased in with conservative thresholds that limit the number of false positives and highlight the potential benefits of these systems. In the long run, these systems increase public confidence by reducing the costs associated with auditing health outcomes through the use of efficient methods and timely surveillance systems. We have no conflicts of interest to declare.
*William Thompson, Lori Hutwagner, Margarette Kolczak National Immunization Program, US Centers for Disease Control and Prevention, Atlanta, GA 30333, USA (e-mail: [email protected]) 1
Farr W. In: Vital statistics, 1885, London: Office of the Sanitary Institute, 1948: 330–33. 2 Farrington CP, Andrews NJ, Beale AD, Catchpole MA. A statistical algorithm for the early detection of outbreaks of infectious disease. J R Stat Soc A 1996; 159: 547–63. 3 Stroup DF, Williamson GD, Herndon JL, Karon J. Detection of aberrations in the occurrence of notifiable diseases surveillance data. Stat Med 1989; 8: 323–29. 4 Evans S. Statistical methods of signal detection. In: Mann RD, Andrews EB, eds. Pharmacovigilance. Chichester: Wiley, 2002: 273–79. 5 Chen R. Vaccine risks, perceived and unknown. Vaccine 1999; 17: S41–S46. 6 Hutwagner L, Thompson WW, Seeman GM, Treadwell T. The bioterrorism preparedness and response early aberration reporting system (EARS). J Urban Health 2003; S1: i89–i96. 7 Baker R, Jones DR, Goldblatt P. Monitoring mortality rates in general practice after Shipman. BMJ 2003; 326: 274–76. 8 Steiner SH, Cook RJ, Farewell VT, Treasure T. Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics 2000; 1: 441–52. 9 Grigg OA, Farewell VT, Spiegelhalter DJ. Use of risk-adjusted CUSUM and RSPRT charts for monitoring in medical contexts. Stat Meth Med Res 2003; 12: 147–70. 10 McPhillips HA, Davis RL, Marcuse EK, Taylor JA. The rotavirus vaccine's withdrawal and physicians’ trust in vaccine safety mechanisms. Arch Pediatr Adolesc Med 2001; 155: 1051–56.
Embryology revisited See page 477 Most doctors have a dim memory of embryology teaching at medical school. In a four-paper series starting in this week’s Lancet, we have tried to highlight some of the more fascinating findings in what now goes under the rather more trendy name of developmental biology. The first paper by Dian Donnai and Andrew Read discusses the role of the clinician in pinpointing some of the syndromes that have helped to elucidate developmental pathways. The next paper by William Horton describes ways of making animal models that help in understanding complex human syndromes, particularly those that affect the skeleton, and in the next paper Frances Goodman details the pathways involved in body patterning and how comparisons between species have been so revealing. The final paper, by our adviser on the series, Judith Hall, discusses the complex events involved in twinning, and brings the topic right back to the clinic. Virginia Barbour The Lancet, London NW1 7BY, UK
THE LANCET • Vol 362 • August 9, 2003 • www.thelancet.com
For personal use. Only reproduce with permission from The Lancet