Review of methods used to measure comorbidity in cancer populations: No gold standard exists

Journal of Clinical Epidemiology 65 (2012) 924e933

REVIEW ARTICLE

Review of methods used to measure comorbidity in cancer populations: No gold standard exists Diana Sarfati* Cancer Control and Screening Research Group, Department of Public Health, University of Otago, PO Box 7343, Wellington South 6242, New Zealand Accepted 16 February 2012; Published online 26 June 2012

Abstract Objective: This article reviews methods used to measure comorbidity in the context of cancer; summarizing methods, identifying contexts in which they have been used, and assessing the validity, reliability, and feasibility of each approach. Study Design and Setting: Studies describing methods to measure comorbidity in epidemiological studies related to cancer were identified. Data relating to content, face, and criterion validity, reliability, and feasibility were collected. Results: Two thousand nine hundred seventy-five abstracts were reviewed and 21 separate approaches identified. Content and face validity varied but tended to be higher for measures developed for cancer populations. Some evidence supporting criterion validity of all approaches was found. Where reported, reliability tended to be moderate to high. Some approaches tended to score well on all aspects but were resource intensive in terms of data collection. Eight indices scored at least moderately well on all criteria, three of which demonstrated usefulness in relation to non-site specific cancer (Charlson Comorbidity Index, Elixhauser approach, and National Cancer Institute [Combined] Index). Conclusions: No gold standard approach to measuring comorbidity in the context of cancer exists. Approaches vary in their strengths and weaknesses, with the choice of measure depending on the study question, population studied, and data available. Ó 2012 Elsevier Inc. All rights reserved. Keywords: Comorbidity; Neoplasms; Multimorbidity; Measurement; Validity; Reliability

1. Introduction As populations age, the prevalence of chronic disease increases. Almost all chronic diseases are more common among the elderly than younger adults, and many of these conditions are not life threatening in the short term. Consequently, many people live with, rather than die from chronic health conditions. Cancer is often a chronic disease and is itself more prevalent among the elderly. Concomitant chronic disease in addition to cancer is now, therefore, the norm rather than the exception, and it can have a profound effect on individuals [1e3]. Comorbidity results in increased risk of hospitalization, adverse effects of treatment, multiple competing demands on both patient and health care professionals, high health care costs, reduced quality of life, and higher mortality [3e12]. Health care service providers, policy makers, and researchers need to be able to respond adequately to the requirements of individuals with complex health needs [13].

* Corresponding author. Tel.: þ64-4-918-6042; fax: þ64-4-389-5319. E-mail address: [email protected] 0895-4356/$ - see front matter Ó 2012 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2012.02.017

Despite the importance of comorbidity in the care of cancer patients, there is no consensus about how to define it and even less on how to measure it in the context of policy, planning, or research. The difficulties in measuring comorbidity arise from many factors, one of which is that the definition of comorbidity depends on the definition of the primary condition. For example, different conditions are likely to be important in terms of their impact on outcomes for patients with breast cancer compared with those with congestive heart failure. For this reason, a number of authors have suggested that disease-specific indices, such as cancer-specific measures are preferable to general ones [14e16]. However, whereas generic comorbidity scales may lack specificity, individualized comorbidity indices lack comparability across research settings. The few studies that reviewed measurement of comorbidity have examined a subset of indices [1,2,17,18], focused on methodological issues [19], and/or reviewed indices in general rather than cancer-specific measures [14]. Over the last few years there have been an increasing number of cancerrelated studies that include measures of comorbidity but little consistency in how this construct is measured. This study reviews approaches to measuring comorbidity in the context of cancer. It summarizes each method, indicates the context in

D. Sarfati / Journal of Clinical Epidemiology 65 (2012) 924e933

925

Table 1. List of abbreviations

What is new? Key findings:  Whereas no gold standard approach to measuring comorbidity in cancer exists, at least 21 approaches have been developed. The choice of approach depends on the study question, data availability, and population studied. What this adds to what was known?  Despite increasing numbers of cancer-related studies that include measures of comorbidity, there is a lack of consensus on how to measure this construct. This study provides a comprehensive review of approaches used to measure comorbidity in the context of cancer. What is the implication and what should change now?  Comorbidity is common and important among cancer patients; therefore, consideration of comorbidity in cancer-related epidemiological studies is crucial. An assessment of best approach given study requirements is necessary.

which each has been used, and assesses the validity, reliability, and feasibility of each approach.

2. Methods A Medline search was carried out for the period 1948 to 31 Dec 2010, using the search terms ‘‘Cancer’’ or ‘‘neoplasms’’ and ‘‘comorbidity,’’ ‘‘multimorbidity,’’ or ‘‘concomitant disease.’’ Studies were limited to those written in English. All abstracts were reviewed by the author, and relevant studies that described methods to measure comorbidity as a confounding, mediating, or explanatory variable in the context of cancer outcomes (survival, treatment receipt, and recurrence) were obtained. Studies that detailed subsequent development or validation of these methods also were obtained. Additional studies were identified from reference lists from articles obtained by the literature search. For a full list of studies cited in this review, see Appendix on the journal’s web site at www.jclinepi.com (Table 1). Data relating to each index or measure were collected in relation to: 1. A general description of the measure or index. This included the original purpose of the index or measure, the process through which comorbid conditions were identified, whether severity was accounted for, whether and how conditions were combined to form an index, whether the index or measure provided ordinal or

ACE-27 ACGs ASA CCI CDS NIA CIRS DCG ICED KFI CPI MACSS NCI SCI TIBI WUHNCI

Adult Comorbidity Evaluation-27 Adjusted Clinical Groups American Society of Anesthesiologists Charlson Comorbidity Index Chronic Disease Score National Institute on Aging Cumulative Illness Rating Scale Diagnostic Cost Group Index of Coexistent Disease KaplaneFeinstein Index Comprehensive Prognostic Index Multipurpose Australian Comorbidity Scoring System National Cancer Institute Simplified Comorbidity Index Total Illness Burden Index Washington University Head and Neck Comorbidity Index

continuous scores, the usual distribution of scores (if relevant), and experience with cancer patients. 2. Content and face validity. Both these measures relate to the degree to which a measure actually evaluates the construct that it purports to measure [20]. Content validity assesses the extent to which a measure includes all relevant items, and face validity assesses the extent to which the measure makes sense, given what is known about the construct and the factors used to measure it. These are qualitative assessments which, in this context, includes the degree to which the measure is relevant to cancer, whether all important conditions are included and how these conditions have been selected, whether other important factors are included, such as severity of conditions, and whether the measure can be ‘‘individualized’’ for specific study purposes. 3. Criterion validity is the extent to which a measure correlates with some other measure of the construct under study [20]. Criterion validity can be either concurrent or predictive. a. Concurrent validity refers to the degree to which the measure correlates with another measure taken at the same time. In relation to comorbidity, this will usually be another validated measure of comorbidity. b. Predictive validity is the extent to which the measure is able to predict future outcomes of interest, such as cancer survival or receipt of treatment. 4. Reliability is ‘‘the extent to which repeated measurements of a stable phenomenon by different people at different times and places get similar results’’ [21]. This is particularly relevant where data for comorbidity measurement are abstracted manually. 5. Feasibility relates to the simplicity, cost, time, and effort required to use the measure. 3. Results Two thousand nine hundred seventy-five abstracts were identified that related to comorbidity and cancer, in which

926

D. Sarfati / Journal of Clinical Epidemiology 65 (2012) 924e933

21 separate approaches to measuring comorbidity were identified. Table 2 summarizes the key characteristics of these approaches, specifically the first relevant article in which each measure or index appears, the population characteristics in which each was developed, the sources of data used, and the method for item generation for each approach. The Elixhauser approach has been included both in its original form [e1], and combined as an index [e2]. Simple comorbidity counts and single conditions have not been included in this table. Table A1 (see Table A1 on the journal’s web site at www.jclinepi.com) summarizes the scoring approaches for each measure of comorbidity, including number of items, severity scale, score range (if relevant), and distribution of each index or measure. 3.1. Description of measures of comorbidity 3.1.1. Counts of individual conditions The simplest approach to measuring comorbidity is to measure the prevalence of individual conditions and to treat them separately and/or to combine them by summing the total number of conditions [7] [e3ee9]. Satariano et al. [e10] developed an index based on the sum of up to seven comorbid conditions that had been found to be associated with worse outcomes for women with breast cancer. Their approach has also been used for patients with colon cancer, modified for use with administrative data, and incorporated into the Comprehensive Geriatric Assessment tool (which also includes measures of functional, performance, depression, and cognitive status) [e10ee15]. The National Institute on Aging/National Cancer Institute Collaborative Study on Comorbidity and Cancer (NIA/NCI SEER study) used data on comorbidity abstracted from medical notes linked to Surveillance Epidemiology and End Results (SEER) program data on incident cancers [e16]. Data were collected on specified conditions, which were further divided based on whether or not the patient was receiving active management. A group of high severity conditions also was later specified [e17]. Elixhauser et al. [e1] identified 30 preexisting conditions among acute care hospital patients that had major effect on short-term patient outcomes using administrative data. Their approach has been used in the context of a number of cancers, including breast [e1], cervical [e18], colon [e19], and prostate cancers [e20]. Elixhauser et al. treated conditions separately or as a count but their approach has subsequently been modified to allow it to be expressed as a (weighted) summary score [e2]. Tammemagi et al. calculated the association of comorbidities identified in computerized medical records with 1-year mortality among lung and breast cancer patients [e21,e22]. They treated conditions separately or as a simple count. The Alcohol-Tobacco Related Comorbidity Index was developed for use specifically with patients diagnosed with head and neck cancers and is a count of up to 11 conditions

associated with alcohol or tobacco use [e23]. Finally, the Multipurpose Australian Comorbidity Scoring System (MACSS) used analysis of linked administrative health data to identify preexisting conditions that were associated with adverse outcomes among hospitalized patients in Western Australia. The final MACSS included 102 conditions, all of which were separately included in models. Further work by the same authors incorporated measures of comorbidity timing and severity to the MACSS system [e24]. MACSS was used in a cohort of patients who had undergone mastectomy for breast cancer [e25]. 3.1.2. Organ or system-based approaches These approaches assess the impact of comorbidity on the function (or dysfunction) of body organs or systems (such as the respiratory, cardiovascular, gastrointestinal, and renal systems). The earliest example of this approach is the Cumulative Illness Rating Scale (CIRS), which rated 13 organ systems according to severity of dysfunction based on clinical data on a five-point scale from 0 (no dysfunction) to 4 (extremely severe dysfunction) [e26]. Scores could be kept separate for each organ system or summed to give a total score. CIRS has been subsequently modified for use in geriatric populations (CIRS-G), geriatric psychiatric populations, and for use with acute conditions [e27ee29]. CIRS has been used to identify the negative impact of comorbidity on cancer survival in general [e30,e31] and for a number of specific cancers, including laryngeal cancer [e32,e33], prostate cancer [e34], and colorectal cancer [e35]. The KaplaneFeinstein Index (KFI ) was developed using data from 188 men diagnosed with diabetes mellitus [e36]. Conditions that may have adversely affected an individual’s life expectancy were identified from patient notes and classified according to severity with grade 1 (slight decompensation of vital systems) to grade 3 (recent full decompensation of vital systems). An overall ranking was assigned based on the severity of the single most severe condition, except where there were two or more conditions in different body systems that had a grade 2 (moderate) severity, in which case the overall score was grade 3 (severe). The KFI has been used in a number of cancer-related studies, both by itself and as a comparison to other indices [e12,e33,e55]. The Index of Coexistent Disease (ICED) combined two dimensions: a measure of comorbid disease severity and a measure of functional impairment [e37] and is a modified version of an earlier comorbidity index [e38,e39]. The severity of dysfunction was assessed for each of 14 organ systems on a five-point scale and then combined with a three-point measure of the degree of physical impairment within 10 functional areas into an overall four-point ordinal scale indicating no, mild, moderate, or severe coexistent disease [e40]. Data are required from clinical notes ideally, including nursing, medical, and laboratory findings. The ICED (or its immediate precursor) has been used for assessment of role of comorbidity in treatment and survival

Table 2. Summary of sources of data for development of measures of comorbidity Index name

Author (year)

Purpose

Population developed

CIRS

Linn et al., 1968 [e26]

?

Clinical notes data

No

Judgment

KFI

Kaplan and Feinstein, 1974 [e36]

188 men with diabetes.

Clinical notes data

No

Judgment

Charlson

Charlson et al., 1987 [e56]

608 general medical patients

Clinical notes data

DCGs

Ash et al., 1989 [e94]

Medicare patients

Administrative data

N/A

Empirical

ACGs

Weiner et al., 1991 [e93] Von Korff et al., 1992; Clark et al., 1995 [e88,e89] Greenfield et al., 1993 [e37]

Measure of physical impairment. Measure of comorbidity among diabetic patients. To develop a ‘‘prognostic taxonomy’’ for comorbid conditions. To predict resource use in HMOs. To predict resource use in HMOs. To predict resource use in HMOs.

16,000 HMO enrollees

Administrative data

No

Empirical

122,911 enrollees in an HMO

Pharmaceutical data

No

Judgment and empirical

356 patients undergoing total hip replacement 936 breast cancer patients.

Clinical notes data

No

Judgment

Clinical notes data

Administrative data

Judgment and empirical

Patient symptom report

No

Judgment and empirical

Clinical notes data

No

Empirical

1,779,167 adult acute care hospital patients

Administrative data

No

Judgment and empirical

848 breast cancer patients

Administrative data

No

Judgment and empirical

14,429 prostate and 7,472 breast cancer patients

Administrative data

No

Judgment and empirical

Surgical patients

Clinical notes data

9,386 head and neck cancer patients

Administrative data

May be obtained from administrative data No

ICED

Satariano

Satariano and Ragland, 1994 [e10]

TIBI/TIBI-CaP

Greenfield et al., 1995; Litwin et al., 2007 [e43,e44] Yancik et al., 1996 [e16]

NIA/NCI Collaborative study

Elixhauser

Elixhauser et al., 1998 [e1]

CPI

Fleming et al., 1999 [e83]

NCI comorbidity Index

Klabunde et al., 2000 and 2007 [e3,e103]

ASA

Reid et al., 2001 [e96]

Alcohol-Tobacco Related Comorbidities Index

Reid et al., 2002 [e23]

To measure impact of comorbidity and physical functioning. To assess comorbidity in breast cancer patients. To measure total burden of disease. To investigate comorbidity burden among older cancer patients. To measure comorbidity using administrative data. To develop site-specific measures of comorbidity for breast and prostate cancers. To measure comorbidity among cancer patients using administrative data. To assess acute operative risk. To assess comorbidity among patients with head and neck cancers.

1,738 general patients and 2894 prostate cancer patients 7,600 cancer patients

Alternative data sources

Administrative data, patient questionnaire

Item generation

Empirical

D. Sarfati / Journal of Clinical Epidemiology 65 (2012) 924e933

CDS/RxRisk

Initial data sources used

N/A Known associations with smoking/alcohol.

927

(Continued )

Judgment and empirical

Tammamagi et al., 2003 and 2005 [e21,e22] Holman et al., 2005 [e25]

Colinet et al., 2005 [e86]

van Walraven et al., 2009 [e2]

Tammamagi

SCI

Elixhauser

Abbreviations: HMO, Health Maintenance Organization; N/A, not applicable.

Administrative data 228,565 adult acute care hospital patients

Picirillo et al., 2003 [e46] ACE-27

MACSS

for breast [7] [e39], prostate [e38,e41,e42], and head and neck cancers [e32,e33]. The Total Illness Burden Index (TIBI) is based on patient report of symptoms and was designed to be a measure of case mix and the impact of poor health on functional status and quality of life [e43]. TIBI has subsequently been adapted specifically for use among men with prostate cancer [e44,e45]. In this instrument (TIBI-CaP), 84 items are included in 11 subdimensions for which severity scores are calculated based on patient symptom report. The subdimensions also are weighted according to greatest expected clinical impact on the patient. A subset of TIBI (the cardiopulmonary index) has also been used to assess patient outcomes among breast cancer patients [7] [e14]. Piccirillo et al. [e46ee48] modified the KFI first into Modified Medical Comorbidity Instrument and then into the Adult Comorbidity Evaluation-27 (ACE-27) index. The purpose was specifically to assess comorbidity in the context of cancer. Cancer registry personnel were trained to collect comorbidity data and define it according to ACE-27 protocols [e49]. Twenty-seven conditions were identified based on previous research and clinical judgment. The ACE-27 system grades specific comorbid conditions into three grades according to severity, which are summarized into an overall rating in the same way as the KFI. ACE-27 has been used in a number of cancer-related studies [e11,e50ee55].

No

Judgment No Clinical notes data 735 patients with lung cancer

No Administrative data

Empirical

No Administrative data

1,155 lung and 906 breast cancer patients. 1,069,770 hospital patients

Empirical

Judgment Administrative data Clinical notes data 11,906 cancer patients

Administrative data Clinical notes data 1,094 head and neck cancer patients Piccirillo et al., 2002 [e85] WUHNCI

To assess comorbidity among patients with head and neck cancers. To assess comorbidity among cancer patients. To assess comorbidity among breast and lung cancer patients. To develop a generalized measure of comorbidity. To assess comorbidity among patients with lung cancer. To combine Elixhauser conditions into index.

Alternative data sources Initial data sources used Population developed Purpose Author (year) Index name

Table 2. Continued

Empirical

D. Sarfati / Journal of Clinical Epidemiology 65 (2012) 924e933

Item generation

928

3.1.3. Weighted indices Weighted indices combine conditions by weighting them according to their relative impact on key outcomes. The Charlson Comorbidity Index (CCI) is the most cited comorbidity index in the literature and the first published example of this approach. It was developed using hospital notes data from a cohort of 604 general medical patients [e56]. Charlson et al. developed a weighted index with the weights being equivalent to the (rounded) adjusted relative risks for 1-year mortality for each condition, with a maximum weight of 6. Conditions with relative risks less than 1.2 were excluded from the index. Algorithms have subsequently been developed by several authors to allow administrative data to be used to calculate Charlson scores [e57ee60]; this approach has been validated in a number of studies [e2,e61ee63], and the index has recently been reweighted using administrative data to account for advances in medical care [e64]. Questionnaires have also been developed to allow the calculation of Charlson scores using patients’self-report [e65,e66]. Other studies have used the Charlson approach but reweighted the index specifically for the outcome under study (e.g., [e67ee69]). The Charlson Index has been used as the basis for other comorbidity indices, most notably the NCI Comorbidity Index, which uses a subset of the same conditions but includes data from both inpatient and outpatient administrative records and uses the beta coefficients (rather than the relative risk) of the association of each condition with 1-year mortality to assign weights [e5]. The CCI has been used in almost every setting, with every cancer, including breast [e56,e70,e71], lung [e72,e73], colorectal

D. Sarfati / Journal of Clinical Epidemiology 65 (2012) 924e933

[e6,e35,e74ee77], urological [e78ee80], cervical [e18], head and neck [e23,e33,e81], and hematological cancers [e82]. Fleming et al. [e83] first developed a ‘‘Comprehensive Prognostic Index (CPI),’’ which combined comorbidity, stage, and age to predict survival among a cohort of patients with breast cancer. The association of each of 28 comorbidity categories with 1-year mortality was assessed, and those with a hazard ratio (HR) O 1.2 (n 5 12) were included in a multivariable model, including interaction terms. They calculated an index using the product of the condition HRs and relevant interaction terms. The same authors used a similar approach to develop a prostate cancerespecific index among 2,931 black men [e84]. Both the Washington University Head and Neck Comorbidity Index (WUHNCI) [22] and the Simplified Comorbidity Index (SCI) [e86,e87] were developed for specific cancer sites (head and neck and lung cancer, respectively). Both assessed the impact of specified conditions on mortality and combined them by summing weights based on beta coefficients from multivariable models. The Chronic Disease Score (CDS) and later RxRisk index used population-based pharmaceutical data to measure the chronic disease status of a population [e88]. Medications that are used to treat chronic conditions were identified from data from a large Health Maintenance Organization and assigned a score based on the pattern of medication use. The score was initially based on clinical experience and later on empirically derived severity weights [e89]. A CDS for each individual was calculated by summing the scores assigned for each class of medications using data over a 1-year period. Fishman et al. [e90] further modified the CDS and renamed it the RxRisk Model. The CDS and RxRisk scores have not been used extensively in epidemiological studies among cancer populations [e33,e34,e86,e87]. 3.1.4. Case-mix approaches Case-mix approaches were developed largely in the United States in response to the need to allocate health care resources in managed care environments where populations were enrolled in health care organizations and are not strictly speaking comorbidity indices per se. However, they have been used as a proxy measure of comorbidity among cancer patients [e19]. These systems focus on conditions or categories of conditions that are associated with increased health service utilization or health care costs [e91ee93]. Two examples of this approach are the Diagnostic Cost Group (DCG) and Adjusted Clinical Group (ACG) systems, which were developed at Harvard and John Hopkins Universities, respectively [e92ee95]. 3.1.5. Overall measures of physical health status Some authors have used an overall measure of health status as a proxy measure of comorbidity. An example is the American Society of Anesthesiologists’ (ASA) class, which was developed as a preoperative summary measure of risk of perioperative complications [e96]. The ASA

929

score ranges from 1 (healthy) to 6 (brain dead). It has been used as a method of estimating comorbidity in patients with head and neck and prostate cancers [e96ee99]. 3.2. Content and face validity Some of the factors relevant to content and face validity are presented in Table 2 (‘‘Item generation’’) and Table A1 (see Table A1 on the journal’s web site at www.jclinepi. com) (‘‘Items,’’ ‘‘Severity,’’ ‘‘Scoring Method,’’ and ‘‘Score range’’). Authors have used varying approaches to identify relevant conditions based on clinical experience or literature in the area (e.g., CIRS, KFI, ICED, ACE-27), empirical analysis (e.g., Charlson, NIA/NCI Collaborative study, MACSS, and Tammemagi) or both (e.g., Satariano, Elixhauser, CPI, and NCI Comorbidity Index) (Table 2). All indices are likely to capture some elements of comorbidity important to cancer patients. Individual condition counts and weighted indices include a highly variable number of conditions (ranging from 7 to 102). Some conditions are included almost universally, such as cardiac, respiratory, liver and renal conditions and diabetes; others less so. For example, alcohol abuse, obesity, drug abuse, angina, osteoporosis, nonediabetes endocrine disorders, and tuberculosis are included in several indices but not others. The Charlson and Satariano measures include none of these conditions, and only MACSS includes all of them. The extent to which this is important is likely to depend on the primary condition under study and the study questions to be addressed. The content validity of organ-based measures depends on the validity of the criteria used to categorize individuals into severity categories for each organ system (see Table A1 on the journal’s web site at www.jclinepi.com). These criteria should ideally be explicit and kept up to date with medical knowledge and thus the criteria of ACE-27, which also were designed relatively recently and with a cancer focus may have higher content validity than those of KFI and CIRS [17]. The CDS approach using pharmaceutical data will only identify conditions for which regular medications are prescribed and will be subject to utilization bias and provider variation in prescribing habits. However, medication-based indices may address some of the concerns about using administrative databases, such as inaccurate recording of diagnoses and may be more likely to identify conditions managed in the outpatient system. The process for allocating patients into DCGs or ACGs is entirely based on resource consumption and is not specifically designed to measure comorbidity in the context of cancer. Similarly, ASA grade is a useful measure of acute outcomes in the surgical setting but has not been developed for the purpose of measuring comorbidity in a cancer cohort. Another key component of content and face validity relates to the extent to which severity of conditions are combined (if relevant) into a single measure of comorbidity (see Table A1 on the journal’s web site at www.jclinepi. com). Where conditions are added together in a simple

930

D. Sarfati / Journal of Clinical Epidemiology 65 (2012) 924e933

unweighted index, the implicit assumption is made that all conditions are equally important in their relationship to outcomes, which is unlikely to be true [e10,e23]. Organ or system-based approaches tend to use highly simplified scoring systems. For example, KFI and ACE-27 assume that a ‘‘severe’’ rating in any body system is equivalent and that two ‘‘moderate’’ ratings in different systems have a combined effect equivalent to a single ‘‘severe’’ rating [e36,e47,e48]. Weighted systems are designed to account for the fact that different conditions are likely to have different impacts on outcomes. The Charlson Index is the prototype for all subsequent weighted indices, but it is not without its problems [e56]. It assumes that the impact of multiple conditions is additive and that the prognostic importance of a given comorbid condition is the same as it was when the index was developed and is constant regardless of the primary condition [e59,e100ee102]. Subsequent indices have dealt with these issues to a greater or lesser extent. For example, some authors have used coefficients calculated for specific cancer sites as weights rather than adjusted HRs [e3,e85,e86,e103], and Fleming et al. [e83,e84] explicitly investigated the role of common combinations of conditions. 3.3. Criterion validity There is at least some evidence to support the predictive ability of all measures reviewed here (see Table A2 on the journal’s we b site at www.jclinepi.com). The strength of this evidence varies, but six measures (CIRS, CCI, ICED, Elixhauser, NCI Combined, and ACE-27) have particularly strong evidence of predictive ability in the context of cancer patient outcomes. Table A2 summarizes the findings of 26 studies that have compared the performance of the various measures of comorbidity. There is little consistency between the findings, which depend on various factors, including the size of the study population and the type of cancer studied, the way the indices were categorized, and the outcome measures used. In terms of concurrent validity, measures of comorbidity tend to be correlated with each other, although the strength of that correlation is highly variable [7] [e14,e23,e33, e35,e47,e81,e86,e96,e106]. For example, Silliman et al. [e14] found that the correlations between CCI, TIBI, and the Satariano index were all significant but the correlation coefficients varied between 0.45 and 0.87. All measures of comorbidity mismeasure the underlying construct of comorbidity to some extent, so the degree of correlation between two measures is more related to how closely they relate methodologically than whether one or the other more accurately captures the construct. 3.4. Reliability Reliability is most relevant for indices that use data abstracted from medical records or from patients themselves. The reliability of measures tends to depend on the

simplicity, clarity, and ease of use of the scale, as well as the quality of the data and training of the abstractors. Studies examining the reliability of data collected for CIRS [2,21] [e4,e29,e33,e34,e38,e105,e108], KFI [e12,e109], CCI [2,21] [e12,e109,e110], ICED [21] [e38ee41], and ACE-27 [e59] have all found moderate-to-high levels of interrater reliability. No data were available on the reliability of TIBI, NIA/NCI Collaborative Study Index, or SCI. Interrater reliability is less relevant for the remaining measures because they use administrative data abstracted in a standard manner. However, data obtained from administrative databases is likely to be less accurate and complete than data collected specifically to measure comorbidity. 3.5. Feasibility Indices that require access to clinical notes and training for abstractors are more time consuming to use. For example, Waite et al. reported that it took abstractors a mean of 8.9 min per set of notes to abstract data to calculate a KFI score, 5.9 min for Charlson based on medical notes, and 9.5 min for ICED [e109]. Generally both CIRS and ICED are considered less easy than using CCI [2] [e40], although in one study of five comorbidity indices, CIRS was rated as second best for ease of use (and better than CCI) [e34]. Special training also is required for ACE-27, and the authors reported that the time required to obtain these data was minimal (2.1 min) [e59]. However, other studies have reported longer abstraction time, averaging 16.8 min per person in a cohort of patients with head and neck cancers [e81]. It has been estimated that the TIBI-CaP can be completed by a patient within 15 min [e43,e111]. Whereas measures based on administrative data do not require primary data collection, these databases are often large and unwieldy and require expertise to manage them. Indices with large numbers of individual variables (e.g., MACSS with 102 variables) require large data sets to be statistically feasible.

4. Discussion This study summarized a review of methods used to measure comorbidity in the context of cancer-related epidemiological studies. Although all these approaches aim to measure the same underlying construct, they vary in terms of the purpose for which the measures were developed, whether they are based on individual conditions or organ systems, and the type and detail of data required for their estimation. Additionally, they vary in their ease of use and complexity of design. What is clear from this review is that none of these approaches can be considered clearly superior to the others. Table 3 provides some qualitative criteria to assess each measure of comorbidity in the context of cancer. Although the criteria are highly simplified, they provide a basic framework to compare the various

to feasibility

to reliability

Measure is easy and cheap to use. Uses routinely collected administrative data.

Evidence for poor reliability only.

Measure is time consuming and/or expensive to use. Requires access to medical notes or patient interview data.

Reliability

Feasibility

Predictive validity

Concurrent validity

Content and face validity

***

to predictive

Some evidence to support concurrent validity. Some evidence to support predictive validity. Evidence for moderate level of reliability. Measure is moderately easy and cheap to use. Requires special analysis of administrative data or specialized software.

Used in limited way with cancer patients. One or two sites only. Most relevant items likely to be included. Some assumptions may not be reasonable.

** *

Not generally used for cancer patient populations. Developed among non-cancer patients. Some relevant items likely to be excluded and/or unreasonable scoring assumptions made. Evidence against concurrent validity. Evidence against predictive validity. Experience with cancer patients

Criteria

Table 3. Qualitative criteria used to assess measures of comorbidity in the context of cancer patients

Used extensively among cancer patient populations. All relevant items likely to be included. Reasonable scoring assumptions made. Developed among cancer patient populations. Strong evidence to support concurrent validity. Strong evidence to support predictive validity. Evidence for high level of reliability.

NR

No evidence relating validity found. No evidence relating validity found. No evidence relating found. No evidence relating found.

to concurrent

D. Sarfati / Journal of Clinical Epidemiology 65 (2012) 924e933

931

approaches. Table 4 provides the assessment of 20 measures of comorbidity in the context of cancer, though it should be noted that the outcomes of this assessment may well differ if specific research questions or contexts were considered. For example, if clinical data are already collected, the feasibility of clinical notesebased indices will be scored higher. Similarly, the score for cancer sitee specific indices (e.g., WUHNC and SCI indices) are only relevant for studies of the site specified. There is some evidence to support the predictive validity of all approaches. For all indices, where these criteria were relevant and data could be found, there was at least moderate evidence for concurrent validity and reliability. There was more variability in the remaining three criteria: experience with cancer patients, content and/or face validity, and feasibility. There were three administrative-based approaches (DCGs, ACGs, and MACSS) that have not generally been used in the context of cancer patients, so there is relatively little evidence on their validity in this particular context and these approaches tended to rate lower on content and face validity also. Other indices that rated lower on this criterion were those developed for purposes other than assessing the impact of comorbidity on patient outcome. For example, DCGs, ACGs, and RxRisk were all developed as predictors of resource use, ASA was developed to predict acute perioperative risk, and TIBI was primarily developed as a measure of case mix. Some indices that scored highly on all other criteria scored low on the feasibility criterion, for example, CIRS, ICED, and ACE-27. Although these are all good measures of comorbidity, they all require special collection of data and therefore may not be available at a population level. Of note, is that there is recent work under way to develop a claims-based version of ACE-27 which if further validated will improve the feasibility of this measure [23]. Eight indices scored at least moderately well on all criteria. These were CCI, Satariano, Elixhauser and Tammemagi approaches, Fleming’s CPI, NCI (Combined) Comorbidity Index, Alcohol-Tobacco Related Comorbidities Index, and the WUHNCI. Several of these have been developed specifically for one or two cancer sites (Satariano, Tammemagi approaches, Fleming’s CPI, Alcohol-Tobacco Related Comorbidities Index, and the WUHNCI), leaving only three indices with generally good properties and clear usefulness in relation to cancer generally (CCI, Elixhauser approach, and NCI [Combined] Index). Of these, the NCI (Combined) Index provides weights that are both cancer specific and derived more recently than the other two indices (particularly CCI). If the collection of comorbidity data from clinical notes is feasible (or a claims-based version is available), other measures could be added to this list, particularly ACE-27 which was developed specifically for cancer patients. The strength of these latter measures is that an estimate of severity of particular conditions can be made. Given the difficulty of identifying a single gold standard measure of comorbidity, other approaches have been

932

D. Sarfati / Journal of Clinical Epidemiology 65 (2012) 924e933

Table 4. Qualitative assessment of validity of indices in relation to cancer patient populations Index name CIRS KFI CCI DCGs ACGs CDS/RxRisk ICED Satariano approach TIBI/TIBI-CaP NIA/NCI Collaborative study Elixhauser approach CPI NCI Comorbidity Index ASA Alcohol-Tobacco Related Comorbidities Index WUHNCI ACE-27 Tammamagi approach MACSS SCI

Experience with cancer patients

Content/face validity

Concurrent validity

Predictive validity

Reliabilitya

Feasibilityb

*** ** *** * * ** ** ** ** *** *** ** *** ** **

** ** ** * * * ** ** * ** ** *** ** * **

*** *** *** NR NR ** *** ** *** NR NR NR ** ** **

*** ** *** ** ** ** *** ** ** ** *** ** *** ** **

** ** ** NA NA NA ** *** NR NR NA NA NA ** NA

* * *** ** ** ** * *** * * *** ** *** ** ***

** *** ** * **

*** ** *** ** **

NR *** ** NR **

** *** ** ** **

** *** NA NA NR

*** * ** ** *

NR, Not reported; NA, not applicable. a Reliability assessed when notes review or patient interview was carried out. b The most simple approach is assessed; for example, if both notes review and administrative data are potential data sources, the latter will be assessed.

suggested. Lash et al. [24] suggested an approach that combines different sources of data into a single model. This approach uses a latent variable approach, which allows separate logistic regression models for each comorbidity measurement to be merged into single regression equation [25]. This allows an overall assessment of the impact of comorbidity on the outcome of interest and the roles of each individual index. However, Lash et al. concluded that ‘‘when a single comorbidity index applies directly to a research question, or such an index can be developed, then that single index should be given preference [over multiple informants approaches]’’ [23]. This review has focused on comorbidity, but there are many other related but distinct constructs [10]. For example, functional status is measured by the ability or otherwise to carry out such tasks and is often related to both the presence and the consequences of chronic disease [26]. However, the relationship between these concepts is complex [27e29], and although some indices conflate these two concepts (e.g., ICED and ASA), most treat them separately. The strength of this review is that it is comprehensive and includes measures of comorbidity that have been developed recently. However, only a brief overview of these approaches is possible in this study, and because no gold standard measure of comorbidity exists, no absolute recommendations are possible. A few general guidelines can be considered, however. If a study is focused on a particular cancer site, an index that has been developed specifically for that site may be the most appropriate. If only administrative data are available, CCI, Elixhauser, and NCI

combined indices may be the best options. Of these the NCI Combined Index may be the most valid for cancer patients because of its cancer-specific weighting. If clinical data are available, ACE-27 would all be a good option. If more than one measure of comorbidity is available, a multiple informants approach may be useful. Ultimately, the critical issue is that comorbidity is considered in some way in cancer-related studies. To decide which approach is optimal, researchers should ensure that they are familiar with the strengths and weaknesses of each method, consider their study question(s), the population studied, and the data and resources available before making a final decision.

Acknowledgments The author thanks Professor Peter Crampton, Professor Neil Pearce, Dr. Kristie Carter, and Dr. Beeteng Lim for helpful comments on earlier drafts of this manuscript. This work was funded by the Health Research Council of New Zealand (grant number 10/404).

Appendix Supplementary material Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.jclinepi. 2012.02.017.

D. Sarfati / Journal of Clinical Epidemiology 65 (2012) 924e933

References [1] Extermann M. Measurement and impact of comorbidity in older cancer patients. Crit Rev Oncol Hematol 2000;35:181e200. [2] Extermann M. Measuring comorbidity in older cancer patients. Eur J Cancer 2000;36:453e71. [3] Satariano WA, Silliman RA. Comorbidity: implications for research and practice in geriatric oncology. Crit Rev Oncol Hematol 2003;48:239e48. [4] Feinstein A. The pre-therapeutic classification of co-morbidity in chronic disease. J Chronic Dis 1970;23:455e69. [5] Fortin M, Bravo G, Hudon C, Lapointe L, Almirall J, Dubois MF, et al. Relationship between multimorbidity and health-related quality of life of patients in primary care. Qual Life Res 2006;15:83e91. [6] Gijsen R, Hoeymans N, Schellevis FG, Ruwaard D, Satariano WA, van den Bos GA. Causes and consequences of comorbidity: a review. J Clin Epidemiol 2001;54:661e74. [7] Mandelblatt JS, Bierman AS, Gold K, Zhang Y, Ng JH, Maserejian N, et al. Constructs of burden of illness in older patients with breast cancer: a comparison of measurement methods. Health Serv Res 2001;36:1085e107. [8] Parekh AK, Barton MB. The challenge of multiple comorbidity for the US health care system. JAMA 2010;303:1303e4. [9] Valderas JM, Starfield B, Roland M. Multimorbidity’s many challenges: a research priority in the UK. BMJ 2007;334:1128. [10] Valderas JM, Starfield B, Sibbald B, Salisbury C, Roland M. Defining comorbidity: implications for understanding health and health services. Ann Fam Med 2009;7:357e63. [11] Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. Milbank Q 1996;74:511e44. [12] Yancik R, Ershler W, Satariano W, Hazzard W, Cohen HJ, Ferrucci L. Report of the national institute on aging task force on comorbidity. J Gerontol A Biol Sci Med Sci 2007;62:275e80. [13] van Weel C, Schellevis F. Comorbidity and guidelines: conflicting interests. Lancet 2006;367:550e1. [14] de Groot V, Beckerman H, Lankhorst GJ, Bouter LM. How to measure comorbidity: a critical review of available methods. J Clin Epidemiol 2003;56:221e9. [15] Grunau GL, Sheps S, Goldner EM, Ratner PA. Specific comorbidity risk adjustment was a better predictor of 5-year acute myocardial infarction mortality than general methods. J Clin Epidemiol 2006;59:274e80.

933

[16] van den Akker M, Buntinx F, Roos S, Knottnerus JA. Problems in determining occurrence rates of multimorbidity. J Clin Epidemiol 2001;54:675e9. [17] Hall SF. A user’s guide to selecting a comorbidity index for clinical research. J Clin Epidemiol 2006;59:849e55. [18] Klabunde CN, Warren JL, Legler JM. Assessing comorbidity using claims data: an overview. Med Care 2002;40:26e35. [19] Lash TL, Mor V, Wieland D, Ferrucci L, Satariano W, Silliman RA. Methodology, design, and analytic techniques to address measurement of comorbid disease. J Gerontol A Biol Sci Med Sci 2007;62:281e5. [20] Streiner D, Norman G. Health Measurement Scales: a practical guide to their development and use. 4th ed. Oxford, UK: Oxford University Press; 2008. [21] Hall SF, Groome PA, Streiner DL, Rochon PA. Interrater reliability of measurements of comorbid illness should be reported. J Clin Epidemiol 2006;59:926e33. [22] Piccirillo JF, Lacy PD, Basu A, Spitznagel EL. Development of a new head and neck cancer-specific comorbidity index. Arch Otolaryngol Head Neck Surg 2002;128:1172e9. [23] Fleming ST, Sabatino SA, Kimmick G, Cress R, Wu X-C, TrenthamDietz A, et al. Developing a claim-based version of the ACE-27 comorbidity index: a comparison with medical record review. Med Care 2011;49:752e60. [24] Lash TL, Thwin SS, Horton NJ, Guadagnoli E, Silliman RA. Multiple informants: a new method to assess breast cancer patients’ comorbidity. Am J Epidemiol 2003;157:249e57. [25] Horton NJ, Laird NM, Zahner G. Use of multiple informant data as a predictor in psychiatric epidemiology. Int J Methods Psychiatr Res 1999;8:6e18. [26] Lash TL, Mor V, Wieland D, Ferrucci L, Satariano WA, Silliman R. Methodology, design and analytic techniques to address measurement of comorbid disease. In: White Paper: The National Institute on Aging’s Comorbidity Taskforce. 2004. [27] Masala C, Petretto DR. From disablement to enablement: conceptual models of disability in the 20th century. Disabil Rehabil 2008;30:1233e44. [28] Nagi SZ. An epidemiology of disability among adults in the United States. Milbank Mem Fund Q Health Soc 1976;54:439e67. [29] Verbrugge LM, Jette AM. The disablement process. Soc Sci Med 1994;38:1e14.