- Email: [email protected]
Generalisation from phase III clinical trials: survival, quality of life, and health economics
THE LANCET
Viewpoint
Generalisation from phase III clinical trials: survival, quality of life, and health economics P M Fayers, D J Hand Inclusion of health-economics assessment in randomised clinical trials (RCTs) is becoming increasingly common, especially in trials for diseases such as cancer. The treatments for such diseases may be expensive, have unpleasant side-effects, and all too often result in small improvement in cure rates or only prolong survival by a short time. Several major organisations that run trials, such as the UK Medical Research Council (MRC) and the European Organization for Research and Treatment of Cancer (EORTC), have a policy of always considering health economics and quality-of-life implications when a new RCT is designed. Undoubtedly, implications for health economics should be considered before advocating that, on the basis of results from an RCT, a new treatment should be adopted as standard. Some treatments are expensive and consume disproportionate amounts of healthcare budgets, and the value of their benefits may be questionable or unproven. However, it is debatable whether an RCT is the appropriate environment for detailed economic appraisal of different therapies. A fundamental question is: how well can we generalise about treatment effects, quality of life findings, and costs that are observed in an RCT? A phase III RCT should provide a scientific experiment in which the randomisation has effectively eliminated the possibility of systematic bias. However, many authors have questioned the extent to which results of clinical trials can be generalised and applied to future, non-trial, patients.1–5 They have suggested that there may be an interaction between the treatment factor and the factor that determines the circumstances in which patients receive the therapy (that is, within an RCT or under nontrial conditions). Patients in clinical trials frequently have better survival and better response to therapy than nontrial patients,6,7 irrespective of whether they receive the control or the experimental therapy. Some of the possible reasons for this finding are: trial eligibility criteria may restrict intake to patients with a good prognosis, such as those with early-stage disease; often there is an element of selection when clinicians decide which patients are healthy enough to receive either of the therapies, and therefore appropriate for randomisation; a disproportionate number of clinicians who participate in trials are from academic or other centres of excellence, or may have access to better facilities and to highly trained
Lancet 1997; 350: 1025–27 Unit for Epidemiology and Clinical Research, Faculty of Medicine, Norwegian University of Science and Technology, N-7005 Trondheim, Norway (Prof P M Fayers CStat), and Statistics Department, Open University, Milton Keynes, UK (Prof D J Hand PhD) Correspondence to: Prof P M Fayers
Vol 350 • October 4, 1997
support staff;6,8,9 patients in a clinical trial are likely to receive carefully defined protocol therapy and undergo rigorous follow-up, whereas patients outside clinical trials may be likely to default from the prescribed regimen or for other reasons receive less than the optimum therapy; publicity or general awareness of the trial may change referral patterns, often with patients with earlier stage disease being detected and referred to the participating hospitals. Thus it is not surprising that, on average, patients in RCTs have better prospects of a favourable outcome. Similarly, the advantage of a new therapy may be optimised in the clinical trial setting. For example, patients in an RCT may be more likely to receive the full recommended dose of chemotherapy, and thus realise the full benefit of therapy. On the other hand, administration of some treatments may improve with experience attained through subsequent regular use. In general, however, the rigour of an RCT is likely to result in a realisation of the full potential of any treatment benefit. Any observed treatment advantage may well be smaller outside the RCT. Patients in RCTs are, overall, likely to have a better outcome than other patients, and if the new treatment proves to be better they may also appear to derive larger benefits from it. What, therefore, can we infer from RCTs? Are they a waste of time? The crucial question concerns generalisability of results from trials. Assume there is a treatment advantage in favour of the new therapy. Although patients in RCTs may gain optimum advantage from the new therapy, in non-trial patients the effects are usually in the same direction but possibly diluted. If a clinical trial indicates that treatment A is better than treatment B, then we would expect A to be, on average, better than B for future patients with similar diagnosis and disease stage. Although it has been suggested that in some trials there may be a clear beneficial effect in one type of patient but a harmful effect in others,5,10,11 fortunately this appears to be an unusual and infrequently reported phenomenon. The essential assumption underlying the ability to generalise results from clinical trials is that the differences subsequently observed in non-trial patients will most likely be in the same direction as those detected within the trial, albeit of different magnitude.12 The results should be meaningful in quality, but not necessarily in quantity: “there is simply no logical basis on which to generalise quantitative results from a trial.”1 Hence, the observed differences should be treated with circumspection, providing only a guide as to what might be achieved with the different therapies. Intention-to-treat analysis is an attempt to obtain results relevant to non-trial patients, but it can only go so far. In many RCTs the outcomes of principal interest are cure, response, or survival, with quality of life being an 1025
THE LANCET
additional and secondary outcome. This does not apply to palliative trials or other trials in which quality of life is of primary importance and may even be the sole outcome of interest. Often, however, the results of an RCT comprise the two separate dimensions of response (or cure or survival or other outcome to therapy) and quality of life. The difficulty with generalisation may be more severe for quality-of-life assessments than for objective responses such as survival, and differences in quality of life seen in a trial may bear little resemblance to the magnitude of differences when patients are treated routinely.13,14 One possible reason for this is that clinical trials may well have additional nursing support, and patients may receive extra attention and care if a novel—and potentially highly toxic—therapy is being used and if the side-effects are still being assessed. On the other hand, although academic hospitals may have efficient and well-staffed systems for patients’ care, some of the smaller hospitals that less frequently participate in RCTs may offer a more personal and intimate surrounding for care. These differences may influence patients’ reported quality of life. The placebo effect may also distort both the overall level of quality of life and the apparent treatment differences.15 Thus there are reasons to expect that reported quality of life may differ during and within a clinical trial, when compared with after and outside the trial.13,14 Outcomes such as quality of life may be more readily and strikingly influenced by an RCT setting than the more objective outcomes such as survival. If quality of life varies so much in different settings, is it worth assessing? Again, there is the implicit assumption that any significant differences detected between treatments will continue to be in the same direction, and that the conclusions can be generalised broadly. Also, the quality of life, toxicity, and side-effects observed within the trial provide a guide as to what to expect in future patients, and help the medical staff communicate with patients about potential problems. The information also helps with planning the treatment and management of patients—eg, by indicating the need for antiemetics or dose modification. Quality of life may differ considerably inside and outside an RCT. However, in trials of radical therapy one may seek an improvement in, say, survival even at the expense of inferior quality of life. Therefore any treatment-related changes in quality of life are expected to be in a contrary direction to other outcomes. What do we do if we have data on quality of life and survival, both measured imprecisely and with magnitudes that may differ in other settings, and we wish to examine the tradeoff, to decide whether a new treatment is worthwhile? We emphasised the problems of generalising from unidimensional outcomes, but when there are two dimensions, changing in opposing directions, the difficulties of assessment of whether the treatment is worthwhile are considerably greater. Several methods have been proposed for the combination of quality of life and survival into single summary measures. Quality-adjusted life-years (QALYs)16 and quality-adjusted time without symptoms or toxicity scores (Q-TWiST)17 both assume that a year spent in less than perfect health can be equated to a shorter time in perfect health—eg, patients might be asked whether they would prefer to trade-off longer survival or fewer symptoms. When differences are only seen in one of the 1026
dimensions (eg, worse quality of life but no change in survival) or are in the same direction in both dimensions (eg, both quality of life and survival are improved) it is easier to interpret the trial results. Where large differences in one dimension are offset by tiny changes in the other, one may be tempted to draw inferences, although it should be noted that some patients express a willingness to gain seemingly trivial survival benefit at the expense of considerable suffering,18–22 whilst others are not willing to accept aggressive treatment at all.22 In all other cases, it is dangerous to extrapolate from clinical-trial results to obtain a measure of whether a treatment is worthwhile for future patients, since both dimensions may be subject to biases of unknown magnitude. Most trials are designed to test a specific null hypothesis of no treatment effect; they do this efficiently, eliminating bias. Survival trials do not provide bias-free estimates of the magnitude of survival gain that will be seen in future patients, nor do they provide bias-free estimates of quality-of-life differences to be seen in future patients. They most emphatically do not provide bias-free estimates of the trade-off between quality of life and survival. Health economics adds the third dimension—cost. All three dimensions have to be combined for an overall indication of whether a new treatment is worthwhile. But, in this dimension too, there are few grounds for believing that the treatment costs in a clinical trial will resemble subsequent costs. RCTs usually demand close and frequent monitoring of patients, and high compliance with therapy; these are precisely the conditions to increase costs associated with management of patients.23,24 Staff may manage patients with extra caution if the treatment is new. Also, costs associated with managing toxicity are often higher in RCTs, but with experience clinicians learn to minimise and contain side-effects more efficiently.25 Dosage may be modified with subsequent experience. Attitudes, facilities, and need for extended inpatient treatment may vary. New chemotherapy regimens tend to be expensive initially, but once they are accepted as standard, hospitals or regions often negotiate bulk discounts, and cheaper rival products start to be produced.24,25 Therefore, after completion of a trial finding positive advantage to new therapy, one can expect marked and rapid changes in costs. Are estimates of cost obtained from a clinical trial applicable to subsequent patients in other hospitals, in other regions or countries?25 Sensitivity analysis can explore the impact of increasing or decreasing estimates of the various parameters affecting costs, quality of life, and other outcomes.26 If results are sufficiently extreme, the conclusions will be found robust and deemed convincing. Unfortunately it is difficult to know what variations are relevant, and sensitivity analysis has limited value.27 Clearly we are describing phase III RCTs such as those commonly done in, for instance, cancer therapy. Other RCTs may compare routine and conventional therapies or policies of management, and sometimes mimic real life settings. Phase IV trials are also generally of this nature, and their results may be closer to those of future patients. Health-economics assessment seems more appropriate in those contexts. It takes little effort and should be routine to record basic details about cost of therapy, such as number of doses of chemotherapy and number of nights as an inpatient. It would seem prudent to collect details of any expensive antiemetics or other costly support measures. Vol 350 • October 4, 1997
THE LANCET
We contend that most RCTs gain little by recording more-detailed information on costs. On the contrary, many survival and other trials aim to accrue large numbers of patients and any additional demands on participating clinicians to collect extra data can prove detrimental to patient recruitment—unless specific financial and staff resources are made available for the exercise.28 However, if such funding for health-economics assessment is available, it may often be preferable to assess the real costs and treatment benefits as experienced in a routine, non-RCT, environment. Rarely is one testing a null hypothesis of no difference in cost; we do not need an RCT. We are more interested in pragmatic assessment of the actual cost and actual benefits that will apply to future patients. These studies can also elicit individual patient utilities of the value of treatment, which is an essential component of the assessment of whether a treatment is worthwhile, but difficult to incorporate into an RCT without greatly increasing the hospital workload29 and discouraging patient recruitment or compliance. The implications of not only quality of life but also health economics should always be taken into account when designing new trials. We believe the conclusions will often be that the RCT is not the best place for detailed assessment of individual-patient economic costs, and that most trials should only collect minimal data enabling the broadest of cost indications to be reported. Although RCTs have high internal validity and are relatively biasfree for the assessment of the most effective or two or more treatments, their external validity and the estimates of treatment effect, impact upon quality of life, and economic costs may all be questioned.30 Detailed costs may be swamped by the differences between RCTs and real environments. We recommend that separate methods and studies be used to assess economic value and costs. Simple RCT-based infomation about average drug consumption can be combined with survival data to obtain upper estimates of drug costs; often that will suffice. Such information will indicate whether there may be important cost implications that should be monitored and evaluated in future patients. Finally, we believe that where data in two or three dimensions is collected—such as cure rates, quality of life, and costs—publications of RCTs should report those dimensions separately and be cautious about combining them into superficially simple indexes such as quality-adjusted life-years31 or qualityadjusted time without symptoms of toxicity.17 It remains the task of the clinician to apply the information provided to the situation of their patient, within their environment. References 1 2 3 4 5 6
Bailey KR. Generalizing the results of randomized clinical trials. Control Clin Trials 1994; 15: 15–23. Rubins HB. From clinical trials to clinical practice: generalizing from participant to patient. Control Clin Trials 1994; 15; 7–10. Davis CE. Generalizing from clinical trials. Control Clin Trials 1994; 15: 11–14. Wittes J. Introduction: from clinical trials to clinical practice—four papers from a plenary session. Control Clin Trials 1994; 15: 5–6. Rothwell PM. Can overall results of clinical trials be applied to all patients? Lancet 1995; 345: 1616–19. Stiller CA. Centralised treatment, entry to trials and survival. Br J Cancer 1994; 70; 352–62.
Vol 350 • October 4, 1997
7 8
9
10
11 12 13
14
15
16
17
18
19
20
21
22
23 24 25 26
27
28
29 30
31
Stiller CA. Survival of patients with cancer. BMJ 1989; 299: 1058–59. Aass N, Klepp O, Cavallin-Stahl E, et al. Prognosis factors in unselected patients with nonseminomatous metastatic testicular cancer: a multicenter experience. J Clin Oncol 1991; 9: 818–26. Harding MJ, Paul J, Gillis CR, Kaye SB. Management of malignant teratoma: does referral to a specialist unit matter? Lancet 1993; 341: 999–1002. Horwitz RI, Singer BH, Makuch RW, Viscoli CM. Can treatment that is helpful on average be harmful to some patients? A study of the conflicting information needs of clinical inquiry and drug regulation. J Clin Epidemiol 1996; 49: 395–400. Sørenson TIA. Which patients may be harmed by good treatments? Lancet 1996; 348: 351–52. Fayers PM. Can overall results of clinical-trials be applied to all patients? Lancet 1995; 346: 445–56. Hays RD, Sherbourne CD, Bozzette SA. Pharmacoeconomics and quality of life research beyond the randomized clinical trial. In: Spilker B, ed. Quality of life and pharmacoeconomics in clinical trials. 2nd ed. Philadelphia: Lippinncott-Raven, 1996: 155–59. Cunningham WE, Bozzette SA, Hays RD, Kanouse DE, Shapiro MF. Comparison of health-related quality of life in clinical trial and nonclinical trial HIV infected cohorts. Med Care 1995; 33: AS15–25. Bouchet C, Guillemin F, Briancon S. Nonspecific effects in longitudinal studies: impact on quality of life measures. J Clin Epidemiol 1996; 49: 15–20. Drummond MF, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. Oxford: Oxford Medical Publications, 1987. Gelber RD, Goldhirsch A, Cole BF, Wieand HS, Schroeder G, Krook JE. A quality-adjusted time without symptoms or toxicity (Q-TWiST) analysis of adjuvant tradiation therapy and chemotherapy for resectable rectal cancer. J Natl Cancer Inst 1996; 88: 1039–45. Slevin ML, Stubbs L, Plant HJ, et al. Attitudes to chemotherapy: comparing views of patients with cancer with those of doctors, nurses, and general public. BMJ 1990; 300: 1458–60. Brennes RM, Andersen K, Wist EA. Cancer patients, doctors and nurses vary in their willingness to undertake cancer therapy. Eur J Cancer 1995; 31A: 1955–59. O’Connor AM. Effects of framing and level of probability on patients’ preferences for cancer chemotherapy. J Clin Epidemiol 1989; 42: 119–26. Boyd NF, Sutherland HJ, Heasman KZ, Tritchler DL, Cummings BJ. Whose utilities for decision-analysis? Med Decis Making 1990; 10: 58–67. Brundage MD, Davidson JR, Mackillop WJ. Trading treatment toxicity for survival in locally advanced non-small cell lung cancer. J Clin Oncol 1997; 15: 330–40. Drummond MF. Economic analysis alongside clinical trials: Problems and potential. J Rheumatol 1995; 22: 1403–07. Buxton MJ. National Wilms’ tumor study: economic perspective. Monogr Natl Cancer Inst 1955; 19: 27–29. Gulati SC, Bitran JD. Cost-effective analysis: sleeping with an enemy or a friend? J Clin Oncol 1995; 13: 2152–54. Freund DA, Dittus RS. Double-blind, placebo-controlled trial of daunorubicin and cytarabine with or without recombinant human granulocyte colony-stimulating factor in elderly patients with acute myeloid leukaemia: economic evaluation with attention to inpatient and outpatient resource utilization. Monogr Natl Cancer Inst 1995; 19: 37–40. O’Brien BJ, Drummond MF, Labelle RJ, Willan A. In search of power and significance—issues in the design and analysis of stochastic costeffectiveness studies in health-care. Med Care 1994; 32: 150–63. Bennett CL, Smith TJ, George SL, Hillner BE, Fleishman S, Niell HB. Free-riding and the prisoner’s dilemma: problems in funding economic analyses of phase III cancer clinical trials. J Clin Oncol 1995; 13: 2457–63. Drummond MF, O’Brien BJ. Economic analysis alongside clinical trials: practical considerations J Rheumatol 1995; 22: 1418–19. Rittenhouse BE, O’Brien BJ. Threats to the validity of pharmacoeconomic analyses based on clinical trial data. In: Spilker B, ed. Quality of life and pharmacoeconomics in clinical trials. 2nd edn. Philadelphia: Lippincott-Raven, 1996: 1215–23. Spiegelhalter DJ, Gore SM, Fitzpatrick R, Fletcher AE, Jones DR, Cox DR. Quality of life measures in health care–III: resource allocation. BMJ 1992; 305: 1205–09.
1027
Viewpoint
Generalisation from phase III clinical trials: survival, quality of life, and health economics P M Fayers, D J Hand Inclusion of health-economics assessment in randomised clinical trials (RCTs) is becoming increasingly common, especially in trials for diseases such as cancer. The treatments for such diseases may be expensive, have unpleasant side-effects, and all too often result in small improvement in cure rates or only prolong survival by a short time. Several major organisations that run trials, such as the UK Medical Research Council (MRC) and the European Organization for Research and Treatment of Cancer (EORTC), have a policy of always considering health economics and quality-of-life implications when a new RCT is designed. Undoubtedly, implications for health economics should be considered before advocating that, on the basis of results from an RCT, a new treatment should be adopted as standard. Some treatments are expensive and consume disproportionate amounts of healthcare budgets, and the value of their benefits may be questionable or unproven. However, it is debatable whether an RCT is the appropriate environment for detailed economic appraisal of different therapies. A fundamental question is: how well can we generalise about treatment effects, quality of life findings, and costs that are observed in an RCT? A phase III RCT should provide a scientific experiment in which the randomisation has effectively eliminated the possibility of systematic bias. However, many authors have questioned the extent to which results of clinical trials can be generalised and applied to future, non-trial, patients.1–5 They have suggested that there may be an interaction between the treatment factor and the factor that determines the circumstances in which patients receive the therapy (that is, within an RCT or under nontrial conditions). Patients in clinical trials frequently have better survival and better response to therapy than nontrial patients,6,7 irrespective of whether they receive the control or the experimental therapy. Some of the possible reasons for this finding are: trial eligibility criteria may restrict intake to patients with a good prognosis, such as those with early-stage disease; often there is an element of selection when clinicians decide which patients are healthy enough to receive either of the therapies, and therefore appropriate for randomisation; a disproportionate number of clinicians who participate in trials are from academic or other centres of excellence, or may have access to better facilities and to highly trained
Lancet 1997; 350: 1025–27 Unit for Epidemiology and Clinical Research, Faculty of Medicine, Norwegian University of Science and Technology, N-7005 Trondheim, Norway (Prof P M Fayers CStat), and Statistics Department, Open University, Milton Keynes, UK (Prof D J Hand PhD) Correspondence to: Prof P M Fayers
Vol 350 • October 4, 1997
support staff;6,8,9 patients in a clinical trial are likely to receive carefully defined protocol therapy and undergo rigorous follow-up, whereas patients outside clinical trials may be likely to default from the prescribed regimen or for other reasons receive less than the optimum therapy; publicity or general awareness of the trial may change referral patterns, often with patients with earlier stage disease being detected and referred to the participating hospitals. Thus it is not surprising that, on average, patients in RCTs have better prospects of a favourable outcome. Similarly, the advantage of a new therapy may be optimised in the clinical trial setting. For example, patients in an RCT may be more likely to receive the full recommended dose of chemotherapy, and thus realise the full benefit of therapy. On the other hand, administration of some treatments may improve with experience attained through subsequent regular use. In general, however, the rigour of an RCT is likely to result in a realisation of the full potential of any treatment benefit. Any observed treatment advantage may well be smaller outside the RCT. Patients in RCTs are, overall, likely to have a better outcome than other patients, and if the new treatment proves to be better they may also appear to derive larger benefits from it. What, therefore, can we infer from RCTs? Are they a waste of time? The crucial question concerns generalisability of results from trials. Assume there is a treatment advantage in favour of the new therapy. Although patients in RCTs may gain optimum advantage from the new therapy, in non-trial patients the effects are usually in the same direction but possibly diluted. If a clinical trial indicates that treatment A is better than treatment B, then we would expect A to be, on average, better than B for future patients with similar diagnosis and disease stage. Although it has been suggested that in some trials there may be a clear beneficial effect in one type of patient but a harmful effect in others,5,10,11 fortunately this appears to be an unusual and infrequently reported phenomenon. The essential assumption underlying the ability to generalise results from clinical trials is that the differences subsequently observed in non-trial patients will most likely be in the same direction as those detected within the trial, albeit of different magnitude.12 The results should be meaningful in quality, but not necessarily in quantity: “there is simply no logical basis on which to generalise quantitative results from a trial.”1 Hence, the observed differences should be treated with circumspection, providing only a guide as to what might be achieved with the different therapies. Intention-to-treat analysis is an attempt to obtain results relevant to non-trial patients, but it can only go so far. In many RCTs the outcomes of principal interest are cure, response, or survival, with quality of life being an 1025
THE LANCET
additional and secondary outcome. This does not apply to palliative trials or other trials in which quality of life is of primary importance and may even be the sole outcome of interest. Often, however, the results of an RCT comprise the two separate dimensions of response (or cure or survival or other outcome to therapy) and quality of life. The difficulty with generalisation may be more severe for quality-of-life assessments than for objective responses such as survival, and differences in quality of life seen in a trial may bear little resemblance to the magnitude of differences when patients are treated routinely.13,14 One possible reason for this is that clinical trials may well have additional nursing support, and patients may receive extra attention and care if a novel—and potentially highly toxic—therapy is being used and if the side-effects are still being assessed. On the other hand, although academic hospitals may have efficient and well-staffed systems for patients’ care, some of the smaller hospitals that less frequently participate in RCTs may offer a more personal and intimate surrounding for care. These differences may influence patients’ reported quality of life. The placebo effect may also distort both the overall level of quality of life and the apparent treatment differences.15 Thus there are reasons to expect that reported quality of life may differ during and within a clinical trial, when compared with after and outside the trial.13,14 Outcomes such as quality of life may be more readily and strikingly influenced by an RCT setting than the more objective outcomes such as survival. If quality of life varies so much in different settings, is it worth assessing? Again, there is the implicit assumption that any significant differences detected between treatments will continue to be in the same direction, and that the conclusions can be generalised broadly. Also, the quality of life, toxicity, and side-effects observed within the trial provide a guide as to what to expect in future patients, and help the medical staff communicate with patients about potential problems. The information also helps with planning the treatment and management of patients—eg, by indicating the need for antiemetics or dose modification. Quality of life may differ considerably inside and outside an RCT. However, in trials of radical therapy one may seek an improvement in, say, survival even at the expense of inferior quality of life. Therefore any treatment-related changes in quality of life are expected to be in a contrary direction to other outcomes. What do we do if we have data on quality of life and survival, both measured imprecisely and with magnitudes that may differ in other settings, and we wish to examine the tradeoff, to decide whether a new treatment is worthwhile? We emphasised the problems of generalising from unidimensional outcomes, but when there are two dimensions, changing in opposing directions, the difficulties of assessment of whether the treatment is worthwhile are considerably greater. Several methods have been proposed for the combination of quality of life and survival into single summary measures. Quality-adjusted life-years (QALYs)16 and quality-adjusted time without symptoms or toxicity scores (Q-TWiST)17 both assume that a year spent in less than perfect health can be equated to a shorter time in perfect health—eg, patients might be asked whether they would prefer to trade-off longer survival or fewer symptoms. When differences are only seen in one of the 1026
dimensions (eg, worse quality of life but no change in survival) or are in the same direction in both dimensions (eg, both quality of life and survival are improved) it is easier to interpret the trial results. Where large differences in one dimension are offset by tiny changes in the other, one may be tempted to draw inferences, although it should be noted that some patients express a willingness to gain seemingly trivial survival benefit at the expense of considerable suffering,18–22 whilst others are not willing to accept aggressive treatment at all.22 In all other cases, it is dangerous to extrapolate from clinical-trial results to obtain a measure of whether a treatment is worthwhile for future patients, since both dimensions may be subject to biases of unknown magnitude. Most trials are designed to test a specific null hypothesis of no treatment effect; they do this efficiently, eliminating bias. Survival trials do not provide bias-free estimates of the magnitude of survival gain that will be seen in future patients, nor do they provide bias-free estimates of quality-of-life differences to be seen in future patients. They most emphatically do not provide bias-free estimates of the trade-off between quality of life and survival. Health economics adds the third dimension—cost. All three dimensions have to be combined for an overall indication of whether a new treatment is worthwhile. But, in this dimension too, there are few grounds for believing that the treatment costs in a clinical trial will resemble subsequent costs. RCTs usually demand close and frequent monitoring of patients, and high compliance with therapy; these are precisely the conditions to increase costs associated with management of patients.23,24 Staff may manage patients with extra caution if the treatment is new. Also, costs associated with managing toxicity are often higher in RCTs, but with experience clinicians learn to minimise and contain side-effects more efficiently.25 Dosage may be modified with subsequent experience. Attitudes, facilities, and need for extended inpatient treatment may vary. New chemotherapy regimens tend to be expensive initially, but once they are accepted as standard, hospitals or regions often negotiate bulk discounts, and cheaper rival products start to be produced.24,25 Therefore, after completion of a trial finding positive advantage to new therapy, one can expect marked and rapid changes in costs. Are estimates of cost obtained from a clinical trial applicable to subsequent patients in other hospitals, in other regions or countries?25 Sensitivity analysis can explore the impact of increasing or decreasing estimates of the various parameters affecting costs, quality of life, and other outcomes.26 If results are sufficiently extreme, the conclusions will be found robust and deemed convincing. Unfortunately it is difficult to know what variations are relevant, and sensitivity analysis has limited value.27 Clearly we are describing phase III RCTs such as those commonly done in, for instance, cancer therapy. Other RCTs may compare routine and conventional therapies or policies of management, and sometimes mimic real life settings. Phase IV trials are also generally of this nature, and their results may be closer to those of future patients. Health-economics assessment seems more appropriate in those contexts. It takes little effort and should be routine to record basic details about cost of therapy, such as number of doses of chemotherapy and number of nights as an inpatient. It would seem prudent to collect details of any expensive antiemetics or other costly support measures. Vol 350 • October 4, 1997
THE LANCET
We contend that most RCTs gain little by recording more-detailed information on costs. On the contrary, many survival and other trials aim to accrue large numbers of patients and any additional demands on participating clinicians to collect extra data can prove detrimental to patient recruitment—unless specific financial and staff resources are made available for the exercise.28 However, if such funding for health-economics assessment is available, it may often be preferable to assess the real costs and treatment benefits as experienced in a routine, non-RCT, environment. Rarely is one testing a null hypothesis of no difference in cost; we do not need an RCT. We are more interested in pragmatic assessment of the actual cost and actual benefits that will apply to future patients. These studies can also elicit individual patient utilities of the value of treatment, which is an essential component of the assessment of whether a treatment is worthwhile, but difficult to incorporate into an RCT without greatly increasing the hospital workload29 and discouraging patient recruitment or compliance. The implications of not only quality of life but also health economics should always be taken into account when designing new trials. We believe the conclusions will often be that the RCT is not the best place for detailed assessment of individual-patient economic costs, and that most trials should only collect minimal data enabling the broadest of cost indications to be reported. Although RCTs have high internal validity and are relatively biasfree for the assessment of the most effective or two or more treatments, their external validity and the estimates of treatment effect, impact upon quality of life, and economic costs may all be questioned.30 Detailed costs may be swamped by the differences between RCTs and real environments. We recommend that separate methods and studies be used to assess economic value and costs. Simple RCT-based infomation about average drug consumption can be combined with survival data to obtain upper estimates of drug costs; often that will suffice. Such information will indicate whether there may be important cost implications that should be monitored and evaluated in future patients. Finally, we believe that where data in two or three dimensions is collected—such as cure rates, quality of life, and costs—publications of RCTs should report those dimensions separately and be cautious about combining them into superficially simple indexes such as quality-adjusted life-years31 or qualityadjusted time without symptoms of toxicity.17 It remains the task of the clinician to apply the information provided to the situation of their patient, within their environment. References 1 2 3 4 5 6
Bailey KR. Generalizing the results of randomized clinical trials. Control Clin Trials 1994; 15: 15–23. Rubins HB. From clinical trials to clinical practice: generalizing from participant to patient. Control Clin Trials 1994; 15; 7–10. Davis CE. Generalizing from clinical trials. Control Clin Trials 1994; 15: 11–14. Wittes J. Introduction: from clinical trials to clinical practice—four papers from a plenary session. Control Clin Trials 1994; 15: 5–6. Rothwell PM. Can overall results of clinical trials be applied to all patients? Lancet 1995; 345: 1616–19. Stiller CA. Centralised treatment, entry to trials and survival. Br J Cancer 1994; 70; 352–62.
Vol 350 • October 4, 1997
7 8
9
10
11 12 13
14
15
16
17
18
19
20
21
22
23 24 25 26
27
28
29 30
31
Stiller CA. Survival of patients with cancer. BMJ 1989; 299: 1058–59. Aass N, Klepp O, Cavallin-Stahl E, et al. Prognosis factors in unselected patients with nonseminomatous metastatic testicular cancer: a multicenter experience. J Clin Oncol 1991; 9: 818–26. Harding MJ, Paul J, Gillis CR, Kaye SB. Management of malignant teratoma: does referral to a specialist unit matter? Lancet 1993; 341: 999–1002. Horwitz RI, Singer BH, Makuch RW, Viscoli CM. Can treatment that is helpful on average be harmful to some patients? A study of the conflicting information needs of clinical inquiry and drug regulation. J Clin Epidemiol 1996; 49: 395–400. Sørenson TIA. Which patients may be harmed by good treatments? Lancet 1996; 348: 351–52. Fayers PM. Can overall results of clinical-trials be applied to all patients? Lancet 1995; 346: 445–56. Hays RD, Sherbourne CD, Bozzette SA. Pharmacoeconomics and quality of life research beyond the randomized clinical trial. In: Spilker B, ed. Quality of life and pharmacoeconomics in clinical trials. 2nd ed. Philadelphia: Lippinncott-Raven, 1996: 155–59. Cunningham WE, Bozzette SA, Hays RD, Kanouse DE, Shapiro MF. Comparison of health-related quality of life in clinical trial and nonclinical trial HIV infected cohorts. Med Care 1995; 33: AS15–25. Bouchet C, Guillemin F, Briancon S. Nonspecific effects in longitudinal studies: impact on quality of life measures. J Clin Epidemiol 1996; 49: 15–20. Drummond MF, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. Oxford: Oxford Medical Publications, 1987. Gelber RD, Goldhirsch A, Cole BF, Wieand HS, Schroeder G, Krook JE. A quality-adjusted time without symptoms or toxicity (Q-TWiST) analysis of adjuvant tradiation therapy and chemotherapy for resectable rectal cancer. J Natl Cancer Inst 1996; 88: 1039–45. Slevin ML, Stubbs L, Plant HJ, et al. Attitudes to chemotherapy: comparing views of patients with cancer with those of doctors, nurses, and general public. BMJ 1990; 300: 1458–60. Brennes RM, Andersen K, Wist EA. Cancer patients, doctors and nurses vary in their willingness to undertake cancer therapy. Eur J Cancer 1995; 31A: 1955–59. O’Connor AM. Effects of framing and level of probability on patients’ preferences for cancer chemotherapy. J Clin Epidemiol 1989; 42: 119–26. Boyd NF, Sutherland HJ, Heasman KZ, Tritchler DL, Cummings BJ. Whose utilities for decision-analysis? Med Decis Making 1990; 10: 58–67. Brundage MD, Davidson JR, Mackillop WJ. Trading treatment toxicity for survival in locally advanced non-small cell lung cancer. J Clin Oncol 1997; 15: 330–40. Drummond MF. Economic analysis alongside clinical trials: Problems and potential. J Rheumatol 1995; 22: 1403–07. Buxton MJ. National Wilms’ tumor study: economic perspective. Monogr Natl Cancer Inst 1955; 19: 27–29. Gulati SC, Bitran JD. Cost-effective analysis: sleeping with an enemy or a friend? J Clin Oncol 1995; 13: 2152–54. Freund DA, Dittus RS. Double-blind, placebo-controlled trial of daunorubicin and cytarabine with or without recombinant human granulocyte colony-stimulating factor in elderly patients with acute myeloid leukaemia: economic evaluation with attention to inpatient and outpatient resource utilization. Monogr Natl Cancer Inst 1995; 19: 37–40. O’Brien BJ, Drummond MF, Labelle RJ, Willan A. In search of power and significance—issues in the design and analysis of stochastic costeffectiveness studies in health-care. Med Care 1994; 32: 150–63. Bennett CL, Smith TJ, George SL, Hillner BE, Fleishman S, Niell HB. Free-riding and the prisoner’s dilemma: problems in funding economic analyses of phase III cancer clinical trials. J Clin Oncol 1995; 13: 2457–63. Drummond MF, O’Brien BJ. Economic analysis alongside clinical trials: practical considerations J Rheumatol 1995; 22: 1418–19. Rittenhouse BE, O’Brien BJ. Threats to the validity of pharmacoeconomic analyses based on clinical trial data. In: Spilker B, ed. Quality of life and pharmacoeconomics in clinical trials. 2nd edn. Philadelphia: Lippincott-Raven, 1996: 1215–23. Spiegelhalter DJ, Gore SM, Fitzpatrick R, Fletcher AE, Jones DR, Cox DR. Quality of life measures in health care–III: resource allocation. BMJ 1992; 305: 1205–09.
1027