- Email: [email protected]
Predictive modeling for epidural abscess: what we can, can't, and should do about it
The Spine Journal 15 (2015) 102–104
Commentary
Predictive modeling for epidural abscess: what we can, can’t, and should do about it Andrew J. Schoenfeld, MD MSca,*, Rodney A. Hayward, MDb a
Robert Wood Johnson Clinical Scholars Program, Department of Orthopaedic Surgery, University of Michigan, Ann Arbor Veterans Administration Hospital, 2800 Plymouth Rd, Building 10, RM G016, Ann Arbor, MI 48109, USA b Robert Wood Johnson Clinical Scholars Program, Department of Internal Medicine, Division of General Medicine, University of Michigan, 2800 Plymouth Rd, Building 10, RM G016, Ann Arbor, MI 48109, USA Received 18 September 2014; accepted 30 September 2014
COMMENTARY ON: Ju KL, Kim SD, Melikian R, Bono CM, Harris MB. Predicting patients with concurrent noncontiguous spinal epidural abscess lesions. Spine J 2015;15:95–101 (in this issue).
Epidural abscess is a serious spinal condition, whose incidence has risen sharply over the course of the past two decades [1]. The main concerns associated with this spinal infection are its propensity for rapid clinical deterioration and catastrophic sequelae, including sepsis, meningitis, and permanent paralysis [1–4]. Even in the modern period, with robust antibiotics and advanced surgical techniques, the mortality rate among patients with epidural abscess
FDA device/drug status: Not applicable. Author disclosures: AJS: Grants: Robert Wood Johnson Foundation (E, Paid directly to institution). RAH: Grants: VA Health Services Research & Development Service’s Quality Enhancement Research Initiative (QUERI DIB 98-001) (F, Paid directly to institution), Measurement Core of the Michigan Center for Diabetes Translational Research (National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health [P60 DK-20572, to RAH (F, Paid directly to institution). The disclosure key can be found on the Table of Contents and at www. TheSpineJournalOnline.com. Conflict of Interest: Dr. AJS is a Robert Wood Johnson Foundation Clinical Scholar. This work was supported in part by the VA Health Services Research & Development Service’s Quality Enhancement Research Initiative (QUERI DIB 98-001, to RAH) and by the Measurement Core of the Michigan Center for Diabetes Translational Research (National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health [P60 DK-20572, to RAH]. The Robert Wood Johnson Foundation and the Department of Veterans Affairs were not directly involved in manuscript preparation or review. Any opinions expressed herein do not necessarily reflect the opinions of the Robert Wood Johnson Foundation or the Department of Veterans Affairs. * Corresponding author. Robert Wood Johnson Clinical Scholars Program, Department of Orthopaedic Surgery, University of Michigan, Ann Arbor Veterans Administration Hospital, 2800 Plymouth Rd, Building 10, RM G016, Ann Arbor, MI 48109, USA. Tel.: (734) 647-4844; fax: (734) 647-3301. E-mail address: [email protected] (A.J. Schoenfeld) http://dx.doi.org/10.1016/j.spinee.2014.09.029 1529-9430/Published by Elsevier Inc.
can be as high as 20% [1–3]. Surprisingly, given the virulence of this condition and the impact it can have on the lives of patients, relatively few studies are available beyond retrospective case series describing interventions and shortterm outcomes [3,4]. Viewed in this context, the contribution of Ju et al. [5] advances our understanding of the epidemiology of this condition. Yet, due to limitations the authors readily acknowledge, their predictive rules are not ready for full clinical application. Using a dataset spanning two decades and consisting of cases from two tertiary academic centers, Ju and colleagues [5] identified 233 patients for inclusion, 22 of whom had noncontiguous epidural abscesses. Within this sample, the authors determined that concurrent extraspinal infection, erythrocyte sedimentation rate O95 mm/h and more than 7-day delay in presentation were independent risk factors for noncontiguous abscesses [5]. The resulting prognostic tool extends from a score of 0 to 3. Patients with a score of 0 (no risk factors present) are at low risk for noncontiguous abscesses, whereas as many as 73% of those with a score of 3 (all three risk factors present) may have a noncontiguous abscess. Predictive models in medicine have flourished recently, and are often viewed as a means to incorporate evidencebased practice and reduce unwanted variation in the delivery of health care [6,7]. Moreover, the desire for prognostic scores and treatment algorithms that meaningfully inform medical and surgical management is likely to increase in the future, as physicians, hospital administrators, and insurers strive to institute more process measures capable of improving the quality and efficiency of health care.
A.J. Schoenfeld and R.A. Hayward / The Spine Journal 15 (2015) 102–104
When applying a predictive model, the clinician is interested in the predictive score’s internal validity (does it really work in the population studied), stability across setting/population (reproducibility and generalizability), and stability across time (another type of generalizability) [7]. For example, the same prognostic tool should help guide management for a homeless 48-year-old drug abuser, a 63-year-old executive with high cholesterol, and an 84-year-old retiree with poorly controlled diabetes, all of whom may develop noncontiguous epidural abscesses. Regardless of the individual to whom the predictive score is applied, an accurate determination of the need for specific testing should occur [7]. In addition, the score must be the same for all individuals possessing identical risk factors, irrespective of other circumstances surrounding their condition [7]. Finally, these attributes of the predictive instrument must be recurrently demonstrated to a point that statistically exceeds the possibility of chance, so that better medical decisions are achieved by using the tool than would occur if one just relied on the average prevalence in the population. Of course, when developing a prediction model it is important that the risk factors are measured accurately and that all, or almost all, incidents of the disease are identified correctly [7]. However, a predictive model’s generalizability, reproducibility, and validity also hinge directly on the size and representativeness of the sample used in its development [7]. This means that one must have a sufficient number of cases that fairly and proportionately represent the full spectrum of the disease [7]. With this in mind, it is important to recognize the limitations of the risk prediction tool developed by Ju and colleagues [5]. Despite the major undertaking of collecting all cases in two facilities over almost 20 years, the very low incidence of epidural abscesses resulted in only 22 identified cases of noncontiguous abscess. When developing a predictive model, researchers generally need at least 10 cases with the outcome per risk factor evaluated [8]. Despite the authors’ best efforts, their final dataset is likely smaller than required, and therefore, validation of the prediction model in another dataset is necessary. Another limitation, which was similarly beyond the authors’ control, is that it is possible that cases of noncontiguous abscess went undetected. As the authors recognize: ‘‘.it was difficult to establish the precise reason why certain patients with [epidural abscesses] received full spinal imaging, whereas others did not.provider preference was the predominant reason’’ [5]. As a result, numerous patients may have had unrecognized/undiagnosed noncontiguous abscesses over the course of the study period. As these cases went undetected, their clinical characteristics are inaccessible to the authors. Such facts, when considered in light of the already low prevalence of noncontiguous epidural abscess in the source population (less than 10% according to the current study), make it difficult to envision how the predictive model might effectively alter
103
a surgeon’s empiric estimate of prior probability for any given patient. This should not detract from the work of Ju et al. [5], which sheds important light on a heretofore unexplored aspect of epidural abscess management. As the authors point out, their effort is the largest series to date on cases of noncontiguous epidural abscess, and their descriptive statistics alone make a valuable and immediate contribution to patient care. What the spine surgical community can do with a study of this kind is use the results as a means to better understand the nature of noncontiguous epidural abscesses and take note of certain factors that may be associated with their development. In our opinion, what surgeons cannot (and should not) do is embrace the predictive model without further rigorous prospective testing in a representative patient sample. The authors, themselves, speak to this fact in their limitations section. The question remains, then, as to what spinal care practitioners should do moving forward and perhaps even in the interim. With the proliferation of electronic medical records and institutional registries, several natural experiments seem plausible. Foremost, a multicenter collaboration could be created that would pool respective experience with cases of epidural abscess, essentially replicating the current study but on a wider level. This type of endeavor could more effectively investigate variables associated with noncontiguous epidural abscesses, given a larger sample, broader clinical variation and more recent cases. Even better, a prospective, multisite collaboration could use the results of Ju and colleagues [5] (among others) to develop a protocol identifying patients at elevated risk of noncontiguous epidural abscess, who would then be routinely evaluated using pan-spinal MRI. With such a protocol, the possibility of undiagnosed cases of noncontiguous abscesses would be minimized. The barriers to producing better evidence to guide the care for critically important low incident conditions are substantial, but we are entering an era in which those barriers can be largely overcome by the power of ‘‘Big Data’’ if combined with proactive professional collaboration. For now, until an analysis of appropriate magnitude is completed, judicious application of the findings published by Ju et al. [5] may represent the highest level of evidence-based care affordable to patients with epidural abscess.
References [1] Schoenfeld AJ. Spine infections. In: Cannada L, ed. Orthopaedic Knowledge Update 11. Rosemont, IL: American Academy of Orthopaedic Surgeons, 2014:737–47. [2] Darouiche RO. Spinal epidural abscess. N Engl J Med 2006;355: 2012–20. [3] Reihsaus E, Waldbaur H, Seeling W. Spinal epidural abscess: a metaanalysis of 915 patients. Neurosurg Rev 2000;232:175–204. [4] Adogwa O, Karikari IO, Carr KR, Krucoff M, Ajay D, Fatemi P, et al. Spontaneous spinal epidural abscess in patients 50 years of age and
104
A.J. Schoenfeld and R.A. Hayward / The Spine Journal 15 (2015) 102–104
older: a 15-year institutional perspective and review of the literature. J Neurosurg Spine 2014;20:344–9. [5] Ju KL, Kim SD, Melikian R, Bono CM, Harris MB. Predicting patients with concurrent non-contiguous spinal epidural abscess lesions. Spine J 2015;15:95–101. [6] Hayward RA. Moneyball, gambling, and the new cholesterol guidelines. Circ Cardiovasc Qual Outcomes 2014;7:311–4.
[7] Straus SE, Glasziou P, Richardson WS, Haynes RB. Diagnosis and screening. In: Straus SE, Glasziou P, Richardson WS, Haynes RB, eds. Evidence-based medicine: how to practice and teach it. 4th ed. New York, NY: Elsevier, 2011:137–67. [8] Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996;49:1373–9.
Commentary
Predictive modeling for epidural abscess: what we can, can’t, and should do about it Andrew J. Schoenfeld, MD MSca,*, Rodney A. Hayward, MDb a
Robert Wood Johnson Clinical Scholars Program, Department of Orthopaedic Surgery, University of Michigan, Ann Arbor Veterans Administration Hospital, 2800 Plymouth Rd, Building 10, RM G016, Ann Arbor, MI 48109, USA b Robert Wood Johnson Clinical Scholars Program, Department of Internal Medicine, Division of General Medicine, University of Michigan, 2800 Plymouth Rd, Building 10, RM G016, Ann Arbor, MI 48109, USA Received 18 September 2014; accepted 30 September 2014
COMMENTARY ON: Ju KL, Kim SD, Melikian R, Bono CM, Harris MB. Predicting patients with concurrent noncontiguous spinal epidural abscess lesions. Spine J 2015;15:95–101 (in this issue).
Epidural abscess is a serious spinal condition, whose incidence has risen sharply over the course of the past two decades [1]. The main concerns associated with this spinal infection are its propensity for rapid clinical deterioration and catastrophic sequelae, including sepsis, meningitis, and permanent paralysis [1–4]. Even in the modern period, with robust antibiotics and advanced surgical techniques, the mortality rate among patients with epidural abscess
FDA device/drug status: Not applicable. Author disclosures: AJS: Grants: Robert Wood Johnson Foundation (E, Paid directly to institution). RAH: Grants: VA Health Services Research & Development Service’s Quality Enhancement Research Initiative (QUERI DIB 98-001) (F, Paid directly to institution), Measurement Core of the Michigan Center for Diabetes Translational Research (National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health [P60 DK-20572, to RAH (F, Paid directly to institution). The disclosure key can be found on the Table of Contents and at www. TheSpineJournalOnline.com. Conflict of Interest: Dr. AJS is a Robert Wood Johnson Foundation Clinical Scholar. This work was supported in part by the VA Health Services Research & Development Service’s Quality Enhancement Research Initiative (QUERI DIB 98-001, to RAH) and by the Measurement Core of the Michigan Center for Diabetes Translational Research (National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health [P60 DK-20572, to RAH]. The Robert Wood Johnson Foundation and the Department of Veterans Affairs were not directly involved in manuscript preparation or review. Any opinions expressed herein do not necessarily reflect the opinions of the Robert Wood Johnson Foundation or the Department of Veterans Affairs. * Corresponding author. Robert Wood Johnson Clinical Scholars Program, Department of Orthopaedic Surgery, University of Michigan, Ann Arbor Veterans Administration Hospital, 2800 Plymouth Rd, Building 10, RM G016, Ann Arbor, MI 48109, USA. Tel.: (734) 647-4844; fax: (734) 647-3301. E-mail address: [email protected] (A.J. Schoenfeld) http://dx.doi.org/10.1016/j.spinee.2014.09.029 1529-9430/Published by Elsevier Inc.
can be as high as 20% [1–3]. Surprisingly, given the virulence of this condition and the impact it can have on the lives of patients, relatively few studies are available beyond retrospective case series describing interventions and shortterm outcomes [3,4]. Viewed in this context, the contribution of Ju et al. [5] advances our understanding of the epidemiology of this condition. Yet, due to limitations the authors readily acknowledge, their predictive rules are not ready for full clinical application. Using a dataset spanning two decades and consisting of cases from two tertiary academic centers, Ju and colleagues [5] identified 233 patients for inclusion, 22 of whom had noncontiguous epidural abscesses. Within this sample, the authors determined that concurrent extraspinal infection, erythrocyte sedimentation rate O95 mm/h and more than 7-day delay in presentation were independent risk factors for noncontiguous abscesses [5]. The resulting prognostic tool extends from a score of 0 to 3. Patients with a score of 0 (no risk factors present) are at low risk for noncontiguous abscesses, whereas as many as 73% of those with a score of 3 (all three risk factors present) may have a noncontiguous abscess. Predictive models in medicine have flourished recently, and are often viewed as a means to incorporate evidencebased practice and reduce unwanted variation in the delivery of health care [6,7]. Moreover, the desire for prognostic scores and treatment algorithms that meaningfully inform medical and surgical management is likely to increase in the future, as physicians, hospital administrators, and insurers strive to institute more process measures capable of improving the quality and efficiency of health care.
A.J. Schoenfeld and R.A. Hayward / The Spine Journal 15 (2015) 102–104
When applying a predictive model, the clinician is interested in the predictive score’s internal validity (does it really work in the population studied), stability across setting/population (reproducibility and generalizability), and stability across time (another type of generalizability) [7]. For example, the same prognostic tool should help guide management for a homeless 48-year-old drug abuser, a 63-year-old executive with high cholesterol, and an 84-year-old retiree with poorly controlled diabetes, all of whom may develop noncontiguous epidural abscesses. Regardless of the individual to whom the predictive score is applied, an accurate determination of the need for specific testing should occur [7]. In addition, the score must be the same for all individuals possessing identical risk factors, irrespective of other circumstances surrounding their condition [7]. Finally, these attributes of the predictive instrument must be recurrently demonstrated to a point that statistically exceeds the possibility of chance, so that better medical decisions are achieved by using the tool than would occur if one just relied on the average prevalence in the population. Of course, when developing a prediction model it is important that the risk factors are measured accurately and that all, or almost all, incidents of the disease are identified correctly [7]. However, a predictive model’s generalizability, reproducibility, and validity also hinge directly on the size and representativeness of the sample used in its development [7]. This means that one must have a sufficient number of cases that fairly and proportionately represent the full spectrum of the disease [7]. With this in mind, it is important to recognize the limitations of the risk prediction tool developed by Ju and colleagues [5]. Despite the major undertaking of collecting all cases in two facilities over almost 20 years, the very low incidence of epidural abscesses resulted in only 22 identified cases of noncontiguous abscess. When developing a predictive model, researchers generally need at least 10 cases with the outcome per risk factor evaluated [8]. Despite the authors’ best efforts, their final dataset is likely smaller than required, and therefore, validation of the prediction model in another dataset is necessary. Another limitation, which was similarly beyond the authors’ control, is that it is possible that cases of noncontiguous abscess went undetected. As the authors recognize: ‘‘.it was difficult to establish the precise reason why certain patients with [epidural abscesses] received full spinal imaging, whereas others did not.provider preference was the predominant reason’’ [5]. As a result, numerous patients may have had unrecognized/undiagnosed noncontiguous abscesses over the course of the study period. As these cases went undetected, their clinical characteristics are inaccessible to the authors. Such facts, when considered in light of the already low prevalence of noncontiguous epidural abscess in the source population (less than 10% according to the current study), make it difficult to envision how the predictive model might effectively alter
103
a surgeon’s empiric estimate of prior probability for any given patient. This should not detract from the work of Ju et al. [5], which sheds important light on a heretofore unexplored aspect of epidural abscess management. As the authors point out, their effort is the largest series to date on cases of noncontiguous epidural abscess, and their descriptive statistics alone make a valuable and immediate contribution to patient care. What the spine surgical community can do with a study of this kind is use the results as a means to better understand the nature of noncontiguous epidural abscesses and take note of certain factors that may be associated with their development. In our opinion, what surgeons cannot (and should not) do is embrace the predictive model without further rigorous prospective testing in a representative patient sample. The authors, themselves, speak to this fact in their limitations section. The question remains, then, as to what spinal care practitioners should do moving forward and perhaps even in the interim. With the proliferation of electronic medical records and institutional registries, several natural experiments seem plausible. Foremost, a multicenter collaboration could be created that would pool respective experience with cases of epidural abscess, essentially replicating the current study but on a wider level. This type of endeavor could more effectively investigate variables associated with noncontiguous epidural abscesses, given a larger sample, broader clinical variation and more recent cases. Even better, a prospective, multisite collaboration could use the results of Ju and colleagues [5] (among others) to develop a protocol identifying patients at elevated risk of noncontiguous epidural abscess, who would then be routinely evaluated using pan-spinal MRI. With such a protocol, the possibility of undiagnosed cases of noncontiguous abscesses would be minimized. The barriers to producing better evidence to guide the care for critically important low incident conditions are substantial, but we are entering an era in which those barriers can be largely overcome by the power of ‘‘Big Data’’ if combined with proactive professional collaboration. For now, until an analysis of appropriate magnitude is completed, judicious application of the findings published by Ju et al. [5] may represent the highest level of evidence-based care affordable to patients with epidural abscess.
References [1] Schoenfeld AJ. Spine infections. In: Cannada L, ed. Orthopaedic Knowledge Update 11. Rosemont, IL: American Academy of Orthopaedic Surgeons, 2014:737–47. [2] Darouiche RO. Spinal epidural abscess. N Engl J Med 2006;355: 2012–20. [3] Reihsaus E, Waldbaur H, Seeling W. Spinal epidural abscess: a metaanalysis of 915 patients. Neurosurg Rev 2000;232:175–204. [4] Adogwa O, Karikari IO, Carr KR, Krucoff M, Ajay D, Fatemi P, et al. Spontaneous spinal epidural abscess in patients 50 years of age and
104
A.J. Schoenfeld and R.A. Hayward / The Spine Journal 15 (2015) 102–104
older: a 15-year institutional perspective and review of the literature. J Neurosurg Spine 2014;20:344–9. [5] Ju KL, Kim SD, Melikian R, Bono CM, Harris MB. Predicting patients with concurrent non-contiguous spinal epidural abscess lesions. Spine J 2015;15:95–101. [6] Hayward RA. Moneyball, gambling, and the new cholesterol guidelines. Circ Cardiovasc Qual Outcomes 2014;7:311–4.
[7] Straus SE, Glasziou P, Richardson WS, Haynes RB. Diagnosis and screening. In: Straus SE, Glasziou P, Richardson WS, Haynes RB, eds. Evidence-based medicine: how to practice and teach it. 4th ed. New York, NY: Elsevier, 2011:137–67. [8] Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996;49:1373–9.