- Email: [email protected]
Problems in measuring or interpreting change in patient outcomes
Osteoarthritis and Cartilage (2002) 10, 503–505 © 2002 Published by Elsevier Science Ltd on behalf of OsteoArthritis Research Society International doi:10.1053/joca.2002.0805, available online at http://www.idealibrary.com on
1063–4584/02/$35.00/0
International Cartilage Repair Society
Problems in measuring or interpreting change in patient outcomes A. Carr Academic Rheumatology, University of Nottingham, Clinical Sciences Building, City Hospital, Nottingham NG5 1PB, U.K.
Determining the effectiveness of interventions in osteoarthritis (OA) is often based on the measurement of change in outcomes such as pain, disability and quality of life. In calculating and interpreting change in outcomes, the assumption is that any change (greater than the measurement error for the instrument) represents a real change in outcome (i.e. a real reduction in pain or disability or a real increase in quality of life). In a clinical trial such change is largely attributed to the efficacy of the intervention. The fundamental principle underlying this assessment of change is that an individual’s attitude towards a particular construct and their way of calibrating it will remain stable. For example, that a visual analogue pain score (VAS) of 63 mm obtained before treatment will mean the same as a VAS pain score of 63 mm obtained after treatment. However, there is increasing evidence that many patientperceived outcomes are not stable over time but alter in response to adaptation to illness or symptoms, coping strategies, and changes in expectations resulting from experience (of treatment, other illness and social or demographic factors such as age). Indeed, the ability to recalibrate symptoms such as pain, or to change expectations and definitions about what constitutes quality of life are often viewed as desirable attributes in chronic disease, representing successful coping or adaptation to changed circumstances. Why do we recognize, and welcome, dynamism in patient-centered outcomes in clinical practice but ignore it when making assessments of treatment efficacy? One explanation for this apparent discrepancy is that clinical impressions of the patient’s well-being, standardized measures of outcome based on sound psychometric principles and statistical methods used to calculate change in outcome have largely developed as discrete methods of assessment with their own underlying principles, rules and assumptions. Few observers have recognized the importance of their clinical observations about adaptation to the measurement of outcome or the interpretation of statistical analyses of change. There is, however, a growing interest in highlighting and explaining these relationships and in developing statistical methods for measuring change1–3 that have important applications to the assessment of treatment efficacy in arthritis.
Response shift; an explanation for dynamism in patient-centered outcomes The dynamism in patient-centered outcomes represents what is described as response shift. Response shift refers to a change in the meaning of self-evaluation of a particular outcome and can occur as the result of two factors1: (1) A change in the patient’s internal standards of measurement. In other words, a recalibration of their scale for that outcome. For example, a VAS pain score of 63 mm before treatment may equate to a pain score of 88 mm after treatment because the patient’s expectations of pain relief have altered. (2) A redefinition of the outcome by the patient. For example, the symptom described as pain becomes something different or those factors constituting quality of life change.
Evidence for the existence of response shift Response shift explains many of the apparently paradoxical quality of life and disability scores observed in several populations of patients. For example, patients with severe disabilities have reported good or excellent quality of life, despite experiencing significant problems performing daily tasks, being socially isolated and having limited incomes4. Patients having hemodialysis and peritoneal dialysis, who experience a range of serious health problems, were more likely to rate themselves as ‘very happy’ than the general population5 and patients with neoplasms have rated their quality of life in the top quarter of the WHO quality of life questionnaire (higher scores equate to a better quality of life)6. This ‘disability paradox’ can be explained by response shift in that patients may have adapted to their level of disease and recalibrated their quality of life and disability scales or reconceptualized what constitutes quality of life and disability. There is specific evidence for recalibration response shift in functional disability in elderly people with recent health problems7, in quality of life in kidney transplantation8, and in pain scores in patients with rheumatoid arthritis (RA) (AJ Carr, 2001, personal communication). In the study of kidney transplant patients6 mean pretransplant quality of life scores were 5.23 on a 10-point scale. Post-transplantation, these scores had risen to 7. At 5, 12 and 18 months post-transplant, patients were asked to retrospectively rate their quality of life before their transplant and gave it scores of 3.27, 3.14 and 3.05 respectively, all lower than the original pre-transplant score.
Address correspondence to: Alison Carr, PhD MSc, ARC Senior Lecturer in Muscoskeletal Epidemiology, Academic Rheumatology, University of Nottingham, Clinical Sciences Building, City Hospital, Nottingham NG5 1PB, U.K. Tel: +44 (0)115 8404733; Fax: +44 (0)115 8404732; E-mail: [email protected]
503
504
A. Carr: Problems in measuring or interpreting change in patient outcomes
These results suggest that patients’ quality of life scales had been recalibrated by the experience of significantly improved health after transplantation. RA patients were asked to undertake a similar exercise in rating their pain on a 100 mm VAS before and after treatment and to make retrospective baseline pain ratings after treatment (Then-tests). Calculating changes in pain following treatment using the two sets of baseline ratings gives very different estimates of treatment effect. Data from qualitative studies in RA also support the existence of response shift in pain and function. Patients describe recalibration of their assessments of pain and function, influenced by disease activity and duration (AJ Carr et al. 2002, personal communication).
Methods for estimating response shift in assessments of treatment efficacy Response shift is important to evaluations of treatment efficacy based on patient-centered or health status measures because of its potential to inflate or underestimate treatment effects. A number of methods for estimating its effect have been proposed, based on study design and statistical analysis. For example, it has been suggested that individualized rather than standardized measures of patient outcome should be used; that then-tests should be incorporated into study design; that alternative methods of statistical analysis should be used that enable longitudinal data to be examined for statistical trends; and that qualitative data should supplement quantitative data to identify whether ‘corrections’ need to be made for recalibration or reconceptualization of outcomes. All methods have advantages and disadvantages and there is no single recommended approach. The only recommendation would be to use more than one approach where possible to enable some evaluation to be made of the validity of the methods used. The choice of method should be based on: the purpose of the study, the design of the trial (duration, frequency of assessments, sample size) and the resources available.
retrospective then-tests should be performed at the same time as the conventional post-treatment assessments. The assumption is that the then-tests will be based on the same internal calibration and conceptualization of the outcome as the conventional post-treatment assessment. Any difference between the conventional baseline measure and the then-test is assumed to be due to response shift. Whilst the simplicity of this method is attractive, it is limited by the potential problem of recall bias; studies in pain suggest that the memory for chronic conditions can be very inaccurate13. STATISTICAL METHODS
There are a number of statistical methods that can be used to evaluate response shift by analysing longitudinal data for statistical trends. These include covariance or factor analysis, and growth curve analysis. Whilst they can be very useful when used in conjunction with one of the other methods for assessing response bias, alone they are limited by their inability to give direct estimates of the magnitude of recalibration or reconceptualization and by the fact that some (factor analysis) require very large sample sizes. However, their advantage is that they allow several hypotheses about response shift to be tested. Growth curve analysis in particular is being increasingly used to assess response shift in quality of life data. QUALITATIVE RESEARCH
Qualitative methods such as semi-structured interviews or focus groups can be used to provide detailed explanations of the nature and size of response shifts. This can be done either in a direct manner, by asking patients to describe what constitutes quality of life for them or what a pain score of 25 mm and so on means, or indirectly by asking patients to specify their personal goals at different time points. The main disadvantages with these methods are that they are time consuming and labour intensive to perform and analyse. Nevertheless, they can be very useful adjuncts to other methods.
OUTCOME MEASURES
The rationale behind the use of individualized or preference-based outcome measures such as the SEIQoL (Schedule for the Evaluation of Individualized Quality of Life)9, MACTAR10, Disease Repercussion Profile (DRP)11 or Extended Q-TWiST method12 to assess response shift is that they enable the conceptualization of function or quality of life to be assessed for each individual patient and can evaluate changes in individual preference weights. They can be used to assess both recalibration and reconceptualization response shift. Some (MACTAR and DRP) are relatively easy and straightforward to use but others (for example, the interviewer-administered version of SEIQoL) are labour-intensive and time consuming and using the Q-TWiST method in this way requires complex statistical techniques. METHODOLOGICAL TECHNIQUES
The most widely used method for assessing response shift is the then-test. This involves asking patients to make baseline and post treatment assessments and then to make post-treatment, retrospective assessments of baseline health status using the same outcome measure. These
Summary The measurement and interpretation of change in patient-centered outcomes in OA is likely to be significantly confounded by changes in the ways patients calibrate and conceptualize pain, disability, quality of life and so on. This is particularly the case in long-term intervention studies or studies of the natural history of disease. The prevalence and magnitude of response shift in patient-centered outcomes in OA have not been determined but there is evidence from other musculoskeletal and chronic diseases that it occurs, even over relatively short time periods. There are a number of methods that can be used to identify and quantify response shift and inclusion of the most appropriate of these should be considered in all trials of treatment efficacy.
References 1. Schwartz CE, Sprangers MAG. Methodological approaches for assessing response shift in longitudinal health-related quality-of-life assessment. Soc Sci Med 1999;48:1531–48.
Osteoarthritis and Cartilage Vol. 10, No. 7 2. Sprangers MAG, Schwartz CE. Integrating response shift into health-related quality of life research: a theoretical model. Soc Sci Med 1999;48:1507–15. 3. Allison PJ, Locker D, Feine JS. Quality of life: a dynamic construct. Soc Sci Med 1997;45:221–30. 4. Albracht GL, Devlieger PJ. The disability paradox: high quality of life against all odds. Soc Sci Med 1999;48:977–88. 5. Anonymous Editorial. Quality of Life. Lancet 1991; 333:350–1. 6. Skevington S. Measuring quality of life in Britain. Introducing the WHOQOL-100. J Psychosom Res 1999;47:449–59. 7. Daltroy LH, Larson MG, Eaton HM, Phillips CB, Liang MH. Discrepancies between self-reported and observed physical function in the elderly: the influence of response shift and other factors. Soc Sci Med 1999;48:1549–61. 8. Adang EMM, Kootstra G, van Hoeff JP, Merckelbach HLGJ. Do retrospective and prospective quality of life
505
9. 10.
11. 12.
13.
assessments differ for pancreas-kidney transplant recipients? Transplant International 1998;11:11–15. O’Boyle CA, McGee H, Hickey A, O’Malley K, Joyce CRB. Individual quality of life in patients undergoing hip replacement. Lancet 1992;339:1088–91. Tugwell P, Bombardier C, Buchanan W, Goldsmith C, Grace E, Hanna B. The MACTAR Patient Preference Disability Questionnaire—an individualised functional priority approach for assessing improvement in physical disability in clinical trials in rheumatoid arthritis. J Rheumatol 1987;14:446–51. Carr AJ. A patient-centred approach to evaluation and treatment in rheumatoid arthritis. Br J Rheumatol 1996;35:921–32. Schwartz CE, Cole B, Vickrey B, Gelber R. The Q-TwiST approach for assessing health-related quality of life in epilepsy. Quality of Life Research 1995;4:135–41. Erskine A, Morley S, Pearce S. Memory for pain: a review. Pain 1990;41:255–65.
1063–4584/02/$35.00/0
International Cartilage Repair Society
Problems in measuring or interpreting change in patient outcomes A. Carr Academic Rheumatology, University of Nottingham, Clinical Sciences Building, City Hospital, Nottingham NG5 1PB, U.K.
Determining the effectiveness of interventions in osteoarthritis (OA) is often based on the measurement of change in outcomes such as pain, disability and quality of life. In calculating and interpreting change in outcomes, the assumption is that any change (greater than the measurement error for the instrument) represents a real change in outcome (i.e. a real reduction in pain or disability or a real increase in quality of life). In a clinical trial such change is largely attributed to the efficacy of the intervention. The fundamental principle underlying this assessment of change is that an individual’s attitude towards a particular construct and their way of calibrating it will remain stable. For example, that a visual analogue pain score (VAS) of 63 mm obtained before treatment will mean the same as a VAS pain score of 63 mm obtained after treatment. However, there is increasing evidence that many patientperceived outcomes are not stable over time but alter in response to adaptation to illness or symptoms, coping strategies, and changes in expectations resulting from experience (of treatment, other illness and social or demographic factors such as age). Indeed, the ability to recalibrate symptoms such as pain, or to change expectations and definitions about what constitutes quality of life are often viewed as desirable attributes in chronic disease, representing successful coping or adaptation to changed circumstances. Why do we recognize, and welcome, dynamism in patient-centered outcomes in clinical practice but ignore it when making assessments of treatment efficacy? One explanation for this apparent discrepancy is that clinical impressions of the patient’s well-being, standardized measures of outcome based on sound psychometric principles and statistical methods used to calculate change in outcome have largely developed as discrete methods of assessment with their own underlying principles, rules and assumptions. Few observers have recognized the importance of their clinical observations about adaptation to the measurement of outcome or the interpretation of statistical analyses of change. There is, however, a growing interest in highlighting and explaining these relationships and in developing statistical methods for measuring change1–3 that have important applications to the assessment of treatment efficacy in arthritis.
Response shift; an explanation for dynamism in patient-centered outcomes The dynamism in patient-centered outcomes represents what is described as response shift. Response shift refers to a change in the meaning of self-evaluation of a particular outcome and can occur as the result of two factors1: (1) A change in the patient’s internal standards of measurement. In other words, a recalibration of their scale for that outcome. For example, a VAS pain score of 63 mm before treatment may equate to a pain score of 88 mm after treatment because the patient’s expectations of pain relief have altered. (2) A redefinition of the outcome by the patient. For example, the symptom described as pain becomes something different or those factors constituting quality of life change.
Evidence for the existence of response shift Response shift explains many of the apparently paradoxical quality of life and disability scores observed in several populations of patients. For example, patients with severe disabilities have reported good or excellent quality of life, despite experiencing significant problems performing daily tasks, being socially isolated and having limited incomes4. Patients having hemodialysis and peritoneal dialysis, who experience a range of serious health problems, were more likely to rate themselves as ‘very happy’ than the general population5 and patients with neoplasms have rated their quality of life in the top quarter of the WHO quality of life questionnaire (higher scores equate to a better quality of life)6. This ‘disability paradox’ can be explained by response shift in that patients may have adapted to their level of disease and recalibrated their quality of life and disability scales or reconceptualized what constitutes quality of life and disability. There is specific evidence for recalibration response shift in functional disability in elderly people with recent health problems7, in quality of life in kidney transplantation8, and in pain scores in patients with rheumatoid arthritis (RA) (AJ Carr, 2001, personal communication). In the study of kidney transplant patients6 mean pretransplant quality of life scores were 5.23 on a 10-point scale. Post-transplantation, these scores had risen to 7. At 5, 12 and 18 months post-transplant, patients were asked to retrospectively rate their quality of life before their transplant and gave it scores of 3.27, 3.14 and 3.05 respectively, all lower than the original pre-transplant score.
Address correspondence to: Alison Carr, PhD MSc, ARC Senior Lecturer in Muscoskeletal Epidemiology, Academic Rheumatology, University of Nottingham, Clinical Sciences Building, City Hospital, Nottingham NG5 1PB, U.K. Tel: +44 (0)115 8404733; Fax: +44 (0)115 8404732; E-mail: [email protected]
503
504
A. Carr: Problems in measuring or interpreting change in patient outcomes
These results suggest that patients’ quality of life scales had been recalibrated by the experience of significantly improved health after transplantation. RA patients were asked to undertake a similar exercise in rating their pain on a 100 mm VAS before and after treatment and to make retrospective baseline pain ratings after treatment (Then-tests). Calculating changes in pain following treatment using the two sets of baseline ratings gives very different estimates of treatment effect. Data from qualitative studies in RA also support the existence of response shift in pain and function. Patients describe recalibration of their assessments of pain and function, influenced by disease activity and duration (AJ Carr et al. 2002, personal communication).
Methods for estimating response shift in assessments of treatment efficacy Response shift is important to evaluations of treatment efficacy based on patient-centered or health status measures because of its potential to inflate or underestimate treatment effects. A number of methods for estimating its effect have been proposed, based on study design and statistical analysis. For example, it has been suggested that individualized rather than standardized measures of patient outcome should be used; that then-tests should be incorporated into study design; that alternative methods of statistical analysis should be used that enable longitudinal data to be examined for statistical trends; and that qualitative data should supplement quantitative data to identify whether ‘corrections’ need to be made for recalibration or reconceptualization of outcomes. All methods have advantages and disadvantages and there is no single recommended approach. The only recommendation would be to use more than one approach where possible to enable some evaluation to be made of the validity of the methods used. The choice of method should be based on: the purpose of the study, the design of the trial (duration, frequency of assessments, sample size) and the resources available.
retrospective then-tests should be performed at the same time as the conventional post-treatment assessments. The assumption is that the then-tests will be based on the same internal calibration and conceptualization of the outcome as the conventional post-treatment assessment. Any difference between the conventional baseline measure and the then-test is assumed to be due to response shift. Whilst the simplicity of this method is attractive, it is limited by the potential problem of recall bias; studies in pain suggest that the memory for chronic conditions can be very inaccurate13. STATISTICAL METHODS
There are a number of statistical methods that can be used to evaluate response shift by analysing longitudinal data for statistical trends. These include covariance or factor analysis, and growth curve analysis. Whilst they can be very useful when used in conjunction with one of the other methods for assessing response bias, alone they are limited by their inability to give direct estimates of the magnitude of recalibration or reconceptualization and by the fact that some (factor analysis) require very large sample sizes. However, their advantage is that they allow several hypotheses about response shift to be tested. Growth curve analysis in particular is being increasingly used to assess response shift in quality of life data. QUALITATIVE RESEARCH
Qualitative methods such as semi-structured interviews or focus groups can be used to provide detailed explanations of the nature and size of response shifts. This can be done either in a direct manner, by asking patients to describe what constitutes quality of life for them or what a pain score of 25 mm and so on means, or indirectly by asking patients to specify their personal goals at different time points. The main disadvantages with these methods are that they are time consuming and labour intensive to perform and analyse. Nevertheless, they can be very useful adjuncts to other methods.
OUTCOME MEASURES
The rationale behind the use of individualized or preference-based outcome measures such as the SEIQoL (Schedule for the Evaluation of Individualized Quality of Life)9, MACTAR10, Disease Repercussion Profile (DRP)11 or Extended Q-TWiST method12 to assess response shift is that they enable the conceptualization of function or quality of life to be assessed for each individual patient and can evaluate changes in individual preference weights. They can be used to assess both recalibration and reconceptualization response shift. Some (MACTAR and DRP) are relatively easy and straightforward to use but others (for example, the interviewer-administered version of SEIQoL) are labour-intensive and time consuming and using the Q-TWiST method in this way requires complex statistical techniques. METHODOLOGICAL TECHNIQUES
The most widely used method for assessing response shift is the then-test. This involves asking patients to make baseline and post treatment assessments and then to make post-treatment, retrospective assessments of baseline health status using the same outcome measure. These
Summary The measurement and interpretation of change in patient-centered outcomes in OA is likely to be significantly confounded by changes in the ways patients calibrate and conceptualize pain, disability, quality of life and so on. This is particularly the case in long-term intervention studies or studies of the natural history of disease. The prevalence and magnitude of response shift in patient-centered outcomes in OA have not been determined but there is evidence from other musculoskeletal and chronic diseases that it occurs, even over relatively short time periods. There are a number of methods that can be used to identify and quantify response shift and inclusion of the most appropriate of these should be considered in all trials of treatment efficacy.
References 1. Schwartz CE, Sprangers MAG. Methodological approaches for assessing response shift in longitudinal health-related quality-of-life assessment. Soc Sci Med 1999;48:1531–48.
Osteoarthritis and Cartilage Vol. 10, No. 7 2. Sprangers MAG, Schwartz CE. Integrating response shift into health-related quality of life research: a theoretical model. Soc Sci Med 1999;48:1507–15. 3. Allison PJ, Locker D, Feine JS. Quality of life: a dynamic construct. Soc Sci Med 1997;45:221–30. 4. Albracht GL, Devlieger PJ. The disability paradox: high quality of life against all odds. Soc Sci Med 1999;48:977–88. 5. Anonymous Editorial. Quality of Life. Lancet 1991; 333:350–1. 6. Skevington S. Measuring quality of life in Britain. Introducing the WHOQOL-100. J Psychosom Res 1999;47:449–59. 7. Daltroy LH, Larson MG, Eaton HM, Phillips CB, Liang MH. Discrepancies between self-reported and observed physical function in the elderly: the influence of response shift and other factors. Soc Sci Med 1999;48:1549–61. 8. Adang EMM, Kootstra G, van Hoeff JP, Merckelbach HLGJ. Do retrospective and prospective quality of life
505
9. 10.
11. 12.
13.
assessments differ for pancreas-kidney transplant recipients? Transplant International 1998;11:11–15. O’Boyle CA, McGee H, Hickey A, O’Malley K, Joyce CRB. Individual quality of life in patients undergoing hip replacement. Lancet 1992;339:1088–91. Tugwell P, Bombardier C, Buchanan W, Goldsmith C, Grace E, Hanna B. The MACTAR Patient Preference Disability Questionnaire—an individualised functional priority approach for assessing improvement in physical disability in clinical trials in rheumatoid arthritis. J Rheumatol 1987;14:446–51. Carr AJ. A patient-centred approach to evaluation and treatment in rheumatoid arthritis. Br J Rheumatol 1996;35:921–32. Schwartz CE, Cole B, Vickrey B, Gelber R. The Q-TwiST approach for assessing health-related quality of life in epilepsy. Quality of Life Research 1995;4:135–41. Erskine A, Morley S, Pearce S. Memory for pain: a review. Pain 1990;41:255–65.