- Email: [email protected]
The need for validation studies of comorbidity adjustment instruments
Clinical Therapeutics/Volume 31, Number 7,2009
Editorial The Need for Validation Studies of Comorbidity Adjustment Instruments Comorbidity is defined as "any distinct additional entity that has existed or may occur during the clinical course of a pa tient who has the index disease under stud y." 1 Comorbidity adjustment instruments are sta tistical measures used to predict a study outcome or to control for confounds that may result from variation in patients' health status; they are widely used in epidemiologic, outcomes, and health services research. Other frequently used terms referring to these instruments include comorbidity index, comorbidity score, comorbidity measure, severity measure, and risk adjustment tool. Measuring comorbidity as a construct is a challenging task. The number of diseases, for instance, is a variable commonly used to adjust for comorbidity. Common terms such as case mix, risk, or severity may be used when we converse with people in our disciplines, yet it may be difficult to fully grasp their different meanings when we refer to these terms. Iezzoni 2 pointed out that adjusting for risk requires thorough knowledge of the outcome of interest, study time frame, study population, and the purpose of the study. For example, a comorbidity adjustment instrument designed to predict 1-year mortality may not be effective in estimating health care expenditures. 3 Likewise, an instrument derived from a Medicaid patient population may not perform efficiently when setting payment rates for private insurers. Nevertheless, the use of comorbidity adjustment instruments in epidemiologic and health services research has become a standard practice, and numerous instruments have been developed since researchers' early attempts in the 1970s. 4 ,5 Despite the widespread use of comorbidity adjustment instruments in the literature, a major challenge lies in choosing the appropriate instrument from the variety of instruments available. Researchers are often faced with a number of limitations, including unavailability of data, financial constraints, and lack of information about the performance of different instruments. Generally, comorbidity adjustment instruments are either based on clinical diagnoses, such as the codes from the International Classification of Diseases, Ninth Edition, Clinical Modification, or pharmacy data, such as the National Drug Code. Because of differences in the performance of diagnosisand medication-based instruments (depending on the study setting and outcome of interest),6,7 it would be reasonable to choose accordingly between these broad categories of instruments. However, investigators do not necessarily have access to both diagnosis and pharmacy records in their data sets, and decisions may be based largely on what types of data are available. The growing popularity of comorbidity adjustment instruments also means that more instruments, both proprietary and nonproprietary, have become available to researchers. With limited financial resources usually allocated to each research project, it could appear more attractive to simply use an instrument that is free of charge instead of one sold commercially, without sufficient consideration given to the question of whether the selected instrument is the most accurate for the study. Furthermore, despite the growing interest in examining available comorbidity adjustment instruments, not enough information is available about how well each instrument performs by itself in different settings, how well an instrument competes against another, or whether a combination of several instruments outperforms a single instrument. Without the backing of validation studies, the choice of a comorbidity adjustment instrument may be based predominantly on convenience rather than empirical evidence. A growing body of literature has been dedicated to the examination of available comorbidity adjustment instruments 8- 11 ; however, only a few studies have attempted to evaluate these validity studies. Schneeweiss and Maclure 12 assessed the validity evidence for 2 commonly used comorbidity adjustment instruments, the Charlson comorbidity index and the chronic disease score, and concluded that these instruments performed only slightly better than using age adjustment in administrative database analyses. They argued that both poor data quality and the oversimplification of a complex construct such as comorbidity undermine the utility of comorbidity adjustment instruments. As such, the cross-validation of available instruments with a focus on comparative analyses 1578
Volume 31 Number 7
Editorial using identical study populations and outcomes of interest is needed. de Groot et aP examined the validity and reliability evidence for 13 comorbidity adjustment instruments and identified a gap in validity evidence in published literature to support the use of some instruments in clinical research. Although content and predictive validity studies were available for all 13 instruments, data about construct validity, concurrent validity, and reliability were insufficient. 5 Even among studies in which the predictive validity of different instruments was compared within the same population, researchers' opinions varied regarding how an instrument's utility should be evaluated. Most have compared head-to-head individual comorbidity adjustment instruments using improvement in explained variance (ie, R2 change) and the area under a receiver operating characteristic curve (ie, C statistic) 13,14; a few have attempted to compile multiple instruments into a single statistic model, and others have examined the Bayesian and Akaike information criteria and the log-likelihood ratiosY However, it remains inconclusive whether combining several instruments into a single model is statistically appropriate, given that multicollinearity could be an issue. In practice, researchers may also be reluctant to use more than one instrument to control for comorbidity because using several instruments is more time consuming than using just one. Developing a comorbidity adjustment instrument to meet a study's specific objectives allows researchers to use the best-fit model to adjust for comorbidity. However, creating such an instrument is often a lengthy and complicated process. Finding the perfect instrument may not be a realistic goal, but researchers should strive to improve data analyses using the means within their reach. Given the absence of a so-called gold standard, researchers need to choose a validated comorbidity adjustment instrument that is suitable for their study population, outcome of interest, research objective, and data source. In the meantime, more validation studies should be conducted, and a consensus is needed about how to evaluate these instruments. Lung-I Cheng, BPharm Abiola Oladapo, BPharm Karen L. Rascati, BPharm, PhD The University of Texas College of Pharmacy Austin, Texas REFERENCES 1. Feinstein AR. The pre-therapeutic classification of co-morbidity in chronic disease.} Chron Dis. 1970;23:455-468. 2. lezzoni L1. Getting started and defining terms. In: Risk Adjustment for Measuring Health Core Outcomes. 3rd ed. Chicago, III: Health Administration Press; 2003:17-32. 3. Charlson ME, Charlson RE, Peterson jC, et al. The Charlson comorbidity index is adapted to predict costs of chronic disease in primary care patients.} Clin Epidemiol. 2008;61 :1234-1240. 4. Kaplan MH, Feinstein AR. The importance of classifYing co-morbidity in evaluating the outcome of diabetes mellitus.} Chron Dis. 1974;27:387-404. 5. de Groot V, Beckerman H, Lankhorst Gj, Bouter LM. How to measure comorbidity: A critical review of available methods. } Clin Epidemiol. 2003;56:221-229. 6. Wahls TL, Barnett Mj, Rosenthal GE. Predicting resource utilization in a Veterans Health Administration primary care popula-
tion: Comparison of methods based on diagnoses and medications. Med Core. 2004;42:123-128. 7.
McGregor jC, Kim PW, Perencevich EN, et al. Utility of the Chronic Disease Score and Charlson Comorbidity Index as comorbidity measures for use in epidemiologic studies of antibiotic-resistant organisms. Am} Epidemiol. 2005; 161 :483-493.
8. Putnam KG, Buist DS, Fishman P, et al. Chronic disease score as a predictor of hospitalization. Epidemiolog)!. 2002;13: 340-346. 9. van Doorn C, Bogardus ST, Williams CS, et al. Risk adjustment for older hospitalized persons: A comparison of two methods of data collection for the Charlson index.} Clin Epidemiol. 2001 ;54:694-701. 10. Cleves MA, Sanchez N, Draheim M. Evaluation of two competing methods for calculating Charlson's comorbidity index when analyzing short-term mortality using administrative data.} Clin Epidemiol. 1997;50:903-908. 11. Yan Y, Birman-Deych E, Radford Mj, et al. Comorbidity indices to predict mortality from Medicare data: Results from the national registry of atrial fibrillation. Med Care. 2005;43: 1073-1 077.
July 2009
1579
Clinical Therapeutics 12. Schneeweiss S, Maclure M. Use of comorbidity scores for control of confounding in studies using administrative databases. Int } Epidemiol. 2000;29:891-898. 13. Perkins AJ, Kroenke K, Unutzer J, et al. Common comorbidity scales were similar in their ability to predict healthcare costs and mortality.} Clin Epidemiol. 2004;57:1040-1048. 14. Farley JF, Harley CR, Devine JW. A comparison of comorbidity measurements to predict healthcare expenditures. Am} Manag Core. 2006;12:110-119. 15. Baser 0, Palmer L, Stephenson J. The estimation power of alternative comorbidity indices. Value Health. 2008;11 :946-955.
doi:1 0.1 016/j.c1inthera.2009.07.013
1580
Volume 31 Number 7
Editorial The Need for Validation Studies of Comorbidity Adjustment Instruments Comorbidity is defined as "any distinct additional entity that has existed or may occur during the clinical course of a pa tient who has the index disease under stud y." 1 Comorbidity adjustment instruments are sta tistical measures used to predict a study outcome or to control for confounds that may result from variation in patients' health status; they are widely used in epidemiologic, outcomes, and health services research. Other frequently used terms referring to these instruments include comorbidity index, comorbidity score, comorbidity measure, severity measure, and risk adjustment tool. Measuring comorbidity as a construct is a challenging task. The number of diseases, for instance, is a variable commonly used to adjust for comorbidity. Common terms such as case mix, risk, or severity may be used when we converse with people in our disciplines, yet it may be difficult to fully grasp their different meanings when we refer to these terms. Iezzoni 2 pointed out that adjusting for risk requires thorough knowledge of the outcome of interest, study time frame, study population, and the purpose of the study. For example, a comorbidity adjustment instrument designed to predict 1-year mortality may not be effective in estimating health care expenditures. 3 Likewise, an instrument derived from a Medicaid patient population may not perform efficiently when setting payment rates for private insurers. Nevertheless, the use of comorbidity adjustment instruments in epidemiologic and health services research has become a standard practice, and numerous instruments have been developed since researchers' early attempts in the 1970s. 4 ,5 Despite the widespread use of comorbidity adjustment instruments in the literature, a major challenge lies in choosing the appropriate instrument from the variety of instruments available. Researchers are often faced with a number of limitations, including unavailability of data, financial constraints, and lack of information about the performance of different instruments. Generally, comorbidity adjustment instruments are either based on clinical diagnoses, such as the codes from the International Classification of Diseases, Ninth Edition, Clinical Modification, or pharmacy data, such as the National Drug Code. Because of differences in the performance of diagnosisand medication-based instruments (depending on the study setting and outcome of interest),6,7 it would be reasonable to choose accordingly between these broad categories of instruments. However, investigators do not necessarily have access to both diagnosis and pharmacy records in their data sets, and decisions may be based largely on what types of data are available. The growing popularity of comorbidity adjustment instruments also means that more instruments, both proprietary and nonproprietary, have become available to researchers. With limited financial resources usually allocated to each research project, it could appear more attractive to simply use an instrument that is free of charge instead of one sold commercially, without sufficient consideration given to the question of whether the selected instrument is the most accurate for the study. Furthermore, despite the growing interest in examining available comorbidity adjustment instruments, not enough information is available about how well each instrument performs by itself in different settings, how well an instrument competes against another, or whether a combination of several instruments outperforms a single instrument. Without the backing of validation studies, the choice of a comorbidity adjustment instrument may be based predominantly on convenience rather than empirical evidence. A growing body of literature has been dedicated to the examination of available comorbidity adjustment instruments 8- 11 ; however, only a few studies have attempted to evaluate these validity studies. Schneeweiss and Maclure 12 assessed the validity evidence for 2 commonly used comorbidity adjustment instruments, the Charlson comorbidity index and the chronic disease score, and concluded that these instruments performed only slightly better than using age adjustment in administrative database analyses. They argued that both poor data quality and the oversimplification of a complex construct such as comorbidity undermine the utility of comorbidity adjustment instruments. As such, the cross-validation of available instruments with a focus on comparative analyses 1578
Volume 31 Number 7
Editorial using identical study populations and outcomes of interest is needed. de Groot et aP examined the validity and reliability evidence for 13 comorbidity adjustment instruments and identified a gap in validity evidence in published literature to support the use of some instruments in clinical research. Although content and predictive validity studies were available for all 13 instruments, data about construct validity, concurrent validity, and reliability were insufficient. 5 Even among studies in which the predictive validity of different instruments was compared within the same population, researchers' opinions varied regarding how an instrument's utility should be evaluated. Most have compared head-to-head individual comorbidity adjustment instruments using improvement in explained variance (ie, R2 change) and the area under a receiver operating characteristic curve (ie, C statistic) 13,14; a few have attempted to compile multiple instruments into a single statistic model, and others have examined the Bayesian and Akaike information criteria and the log-likelihood ratiosY However, it remains inconclusive whether combining several instruments into a single model is statistically appropriate, given that multicollinearity could be an issue. In practice, researchers may also be reluctant to use more than one instrument to control for comorbidity because using several instruments is more time consuming than using just one. Developing a comorbidity adjustment instrument to meet a study's specific objectives allows researchers to use the best-fit model to adjust for comorbidity. However, creating such an instrument is often a lengthy and complicated process. Finding the perfect instrument may not be a realistic goal, but researchers should strive to improve data analyses using the means within their reach. Given the absence of a so-called gold standard, researchers need to choose a validated comorbidity adjustment instrument that is suitable for their study population, outcome of interest, research objective, and data source. In the meantime, more validation studies should be conducted, and a consensus is needed about how to evaluate these instruments. Lung-I Cheng, BPharm Abiola Oladapo, BPharm Karen L. Rascati, BPharm, PhD The University of Texas College of Pharmacy Austin, Texas REFERENCES 1. Feinstein AR. The pre-therapeutic classification of co-morbidity in chronic disease.} Chron Dis. 1970;23:455-468. 2. lezzoni L1. Getting started and defining terms. In: Risk Adjustment for Measuring Health Core Outcomes. 3rd ed. Chicago, III: Health Administration Press; 2003:17-32. 3. Charlson ME, Charlson RE, Peterson jC, et al. The Charlson comorbidity index is adapted to predict costs of chronic disease in primary care patients.} Clin Epidemiol. 2008;61 :1234-1240. 4. Kaplan MH, Feinstein AR. The importance of classifYing co-morbidity in evaluating the outcome of diabetes mellitus.} Chron Dis. 1974;27:387-404. 5. de Groot V, Beckerman H, Lankhorst Gj, Bouter LM. How to measure comorbidity: A critical review of available methods. } Clin Epidemiol. 2003;56:221-229. 6. Wahls TL, Barnett Mj, Rosenthal GE. Predicting resource utilization in a Veterans Health Administration primary care popula-
tion: Comparison of methods based on diagnoses and medications. Med Core. 2004;42:123-128. 7.
McGregor jC, Kim PW, Perencevich EN, et al. Utility of the Chronic Disease Score and Charlson Comorbidity Index as comorbidity measures for use in epidemiologic studies of antibiotic-resistant organisms. Am} Epidemiol. 2005; 161 :483-493.
8. Putnam KG, Buist DS, Fishman P, et al. Chronic disease score as a predictor of hospitalization. Epidemiolog)!. 2002;13: 340-346. 9. van Doorn C, Bogardus ST, Williams CS, et al. Risk adjustment for older hospitalized persons: A comparison of two methods of data collection for the Charlson index.} Clin Epidemiol. 2001 ;54:694-701. 10. Cleves MA, Sanchez N, Draheim M. Evaluation of two competing methods for calculating Charlson's comorbidity index when analyzing short-term mortality using administrative data.} Clin Epidemiol. 1997;50:903-908. 11. Yan Y, Birman-Deych E, Radford Mj, et al. Comorbidity indices to predict mortality from Medicare data: Results from the national registry of atrial fibrillation. Med Care. 2005;43: 1073-1 077.
July 2009
1579
Clinical Therapeutics 12. Schneeweiss S, Maclure M. Use of comorbidity scores for control of confounding in studies using administrative databases. Int } Epidemiol. 2000;29:891-898. 13. Perkins AJ, Kroenke K, Unutzer J, et al. Common comorbidity scales were similar in their ability to predict healthcare costs and mortality.} Clin Epidemiol. 2004;57:1040-1048. 14. Farley JF, Harley CR, Devine JW. A comparison of comorbidity measurements to predict healthcare expenditures. Am} Manag Core. 2006;12:110-119. 15. Baser 0, Palmer L, Stephenson J. The estimation power of alternative comorbidity indices. Value Health. 2008;11 :946-955.
doi:1 0.1 016/j.c1inthera.2009.07.013
1580
Volume 31 Number 7