Eliciting preferences for alternative drug therapies in oncology: Influence of treatment outcome description, elicitation technique and treatment experience on preferences

J ChronDis Vol. 40, No. 8, pp. 81l-818, 1987 Printedin Great Britain. All rightsreserved

0021-9681187 $3.00+ 0.00 Copyright Q 1987Pergamon Journals Ltd

ELICITING PREFERENCES FOR ALTERNATIVE DRUG THERAPIES IN ONCOLOGY: INFLUENCE OF TREATMENT OUTCOME DESCRIPTION, ELICITATION TECHNIQUE AND TREATMENT EXPERIENCE ON PREFERENCES ANNETTE M. C. O’CONNOR,

NORMAN F. Bow,

and

PADRAIG WARDE, L. STOLBACH

JAMES E. TILL

Instituteof Medical Science, School of Graduate Studies, University of Toronto, and

The Ontario Cancer Institute, Toronto, Ontario, Canada M4X 1K9 (Received in revised form 17 March 1986)

Ah&act-Several methodologic issues arise in eliciting preferences for therapy. Examples are the selection of appropriate descriptions of treatment outcomes and of elicitation techniques. Of particular importance is the correspondence of patients’ anticipated preferences for treatment to actual preferences once they have experienced treatment. Treatment outcome descriptions and elicitation techniques were compared for a hypothetical drug decision problem involving trade-offs between quality and quantity of life. Preferences of 54 cancer patients were elicited before, and 6 weeks following initiation of chemotherapy treatment. Patients’ preferences were not influenced by the way information about side effects was presented, nor the stated probability of survival at high and moderate levels. A riskless rating technique produced different preferences from those of a risky treatment choice method. Although patients experienced significant toxicity following initiation of treatment, their preferences remained stable on retest. The results raise questions about the extent to which patients are willing, at the time of decision making, to trade off survival rate for improved quality of life.

INTRODUCTION

When treatment decisions involve making trade-offs between quality of and quantity of life, the reliability and validity of techniques used to present information and elicit choice or consent from patients is of crucial significance. Important dimensions to consider in eliciting preferences are the way treatment effects are described, the methods of preference elicitation, and the timing of assessment. If descriptions and techniques produce consistent results before and after treatment is initiated, then clinicians can be more confident that the preferences obtained truly represent the patient’s point of view. From previous research, we know that preferences may be sensitive to the way the side effects of treatment are described [l], the elicitation technique that is used [l--3], the level of probability of survival that is presented [l, 4,5] and the treatment experience of the respondent [6].

However, many of these studies were conducted with healthy volunteers in laboratory settings. The few patient studies used different decision contexts with patients who were not involved with the treatment or who were interviewed well after treatment had commenced. In view of the evidence that patients’ and healthy volunteers’ responses differ [7,8] and patients’ responses are more reliable during or shortly after treatment [9], we explored the influence of description of treatment effects, elicitation technique, probability level, and treatment experience on preferences for a group of chemotherapy patients about to begin treatment for the first time. There was little previous information about the influence on preferences of the way side effects of treatment are described to patients. The alternatives are to describe what a “typical” patient might experience, or to attempt to be more realistic and introduce uncertainty of side effects by describing toxicity rates. In the latter approach, it is difficult to establish the proba-

812

ANNETTE M. C. O’CONNORet al.

bility of every possible combination of side effects. Instead, the frequency with which each symptom occurs is described and the treatment outcomes are left to the imagination of the patient. Preliminary work [l] indicated that treatment outcome description had a small but significant effect on preferences for healthy subjects; replication with cancer patients who were receiving treatment was required. Although elicitation techniques such as the rating method and lottery technique have been reasonably reliable measures, the results produced have differed significantly from one another and yielded only fair correlations [l-3]. Such discrepancies are to be expected, since the rating technique measures values when no risk is involved, whereas the lottery technique measures utilities, which are values incorporating attitude toward risk. For this study, both techniques were used to quantify subjects’ tradeoffs between probability of survival and severity of side effects; taken together, they provide a way to compare preferences when risk is involved and when it is not. Use of the lottery technique involves the assumption that preferences obtained at one level of probability of survival (e.g. 100% after one year) will correspond to those obtained at another level of probability (e.g. 60% after one year). Differences in preferences at different probability levels which emerged from a study involving healthy volunteers [l] required verification with cancer patients. A final important issue is the extent to which decisions based on imagined future consequences correspond to those made when patients actually experience the treatment. Do preferences shift with experience, and if so, should the anticipated or actual preference be used in guiding action [9, lo]? Studies on patients living with renal disease [7] or with a colostomy [8] have indicated that patients report higher utilities than non-patients. Two studies using riskless assessment procedures and within-subject rather than between-subject designs have examined the impact of experience on preferences. Laryngeal .. cancer patients’ values for three voice quality dimensions remained stable in spite of significant deterioration in voice quality following radiation therapy 191. In contrast, pregnant womens’ preferences for anaesthesia, obtained one month before and one month after delivery, differed significantly from their preferences during labor [6]. We wished to examine the extent to which results in the present study would corre-

spond to those of patients with laryngeal cancer or would differ from them because of the greater impact of treatment on overall functioning. This study was part of a larger project designed to improve our understanding of methodological problems in eliciting preferences for cancer treatment decisions involving trade-offs between probability of survival and severity of side effects. The long-term goal of the project is to develop improved strategies for explaining treatment decision problems to patients and eliciting preferences. METHODS

The general plan of the study was to develop a series of questions which would describe the side effects of two treatment alternatives as certain and uncertain and would elicit preferences using a riskless rating method and a risky treatment choice approach based on a probability equivalence lottery technique [l]. The treatment choice questions would describe probability of survival as high or moderate. In total, five questions were developed (see Table 1) and asked on two occasions: before and 6 weeks after initiation of chemotherapy treatment. Questions were based on a hypothetical decision problem where trade-offs between side effects and probability of one year survival are made. Although the treatments were identified to patients only as A and B, the side effects were modelled after hormone therapy and chemotherapy (cyclophosphamide, methotrexate, 5 fluorouracil) and represented the most frequently occurring toxicities [ll-151. The trade-off involved choosing between hormone therapy (a pill which produced hot flushes and occasional mild nausea) and chemotherapy (an intravenous treatment which produced nausea, vomiting, fatigue and hair loss) where the probability of one year survival for the less toxic hormone therapy was less than that of chemotherapy. For the riskless rating technique, subjects were presented with descriptions of a chemo-

Table 1. Techniques used to elicit preferences for therapy

Sideeffectscertain

1. Category rating (riskless) 2. Treatment choice 100 (risky, high probability) 3. Treatment choice 60 (risky, moderate probability)

side efeCts ,,,,certai,, 1. Treatment choice 100 (risky, high probability) 2. Treatment choice 60 (risky, moderate probability)

813

Eliciting Preferences Table 2. Treatment outcome descriptions when side effects are certain Treatment A I take a drug by INTRAVENOUS (needle in vein) EVERY THREE WEEKS IN CLINIC. The drug is successful in shrinking the tumor. I have NAUSEA AND VOMITING for 2 DAYS AFTER EACH TREATMENT. I feel TIRED SOME OF THE TIME and avoid my more strenuous household and leisure activities. I have PARTIAL HAIR LOSS and wear a wig when I go out. Otherwise, I feel well. After 1 year of treatment, the tumor remains shrunken.

therapy and hormone therapy remission (see Table 2). They were asked to rate the chemotherapy remission in relation to hormone remission and death by placing a mark on a 100 mm line which was anchored by death and hormone remission. The strength of preference for chemotherapy remission was calculated as the distance between death and chemotherapy remission divided by 100 (e.g. 90 mm/100 mm = 0.90). The risky treatment choice approach quantified preferences by asking subjects to choose between the two unidentified treatments when the one year probabilities of survival for both were equal, (e.g. 100%). After patients chose the less toxic hormone therapy, the chance of survival stayed the same for the toxic chemotherapy (100%) but was systematically lowered for the non-toxic hormone therapy (99%, 95%, 90%) until subjects were indifferent between the two treatments. The probability at which they were indifferent (e.g. 0.90) represented the strength of preference for chemotherapy remission in relation to hormone remission and death [16]. Preferences were assessed beginning at 100% survival for both treatments (Treatment Choice 100) and followed by an assessment beginning at 60% survival (Treatment Choice 60). When assessments were done beginning with 60% survival, the indifference point (e.g. 0.54) was divided by 0.60 to make the values commensurate with those when tradeoffs were elicited at 100% (e.g. 0.54/0.60 = 0.90).

Treatment B I take a PILL DAILY AT HOME. The drug is successful in shrinking the tumor. I have OCCASIONAL NAUSEA from the drug which is usually relieved if I take it with milk. I have OCCASIONAL HOT FLUSHES which can be embarrassing in social occasions. Otherwise, I feel well and can carry out my usual activities. After 1 year of treatment, the tumor remains shrunken.

The order of preference elicitation (Treatment Choice loo/Treatment Choice 60 or Treatment Choice 60/Treatment Choice 100) did not significantly influence the scores which were obtained. Both certain (Table 2) and uncertain descriptions (Table 3) were used for each level of probability. The uncertain description differed from the certain description in that toxicity rates for each side effect were given. Patients were selected from inpatient and outpatient departments of two Metropolitan teaching hospitals. The following inclusion criteria were used in selecting the sample: (1) knows cancer diagnosis, (2) decision to receive chemotherapy for the first time has been made, spoken (3) ambulatory, (4) comprehends English; reads English or French, (5) suitable for evaluating hypothetical drug treatments. Criterion 5 excluded subjects who were mentally or emotionally unable to choose between hypothetical drug treatments. An initial plan to restrict the sample to breast cancer patients was dropped to ensure an adequate sample size in a reasonable period of time. Expansion of the data collection to other diagnostic groups posed no major problems. Most patients stated they had no choice in treatment, and were therefore responding hypothetically. Secondly, most patients identified the chemotherapy description as their treatment and did not identify the hormone therapy. Virtually all preferred the hormones over chemotherapy when the chances of survival for both were equal. The tradeoffs in probability

Table 3. Treatment outcome descriptions when side effects are uncertain Treatment A

Treatment B

Drug A is administered intravenously every three weeks in clinic. The drug is successful in shrinking the tumor. Out of 100 people who take the drug, 70 will experience nausea for two days after each treatment and 30 will have vomiting for two days after each treatment. Seventy will have fatigue some of the time which limits their ability to perform strenuous work, leisure, and household activities and 30 will have partial hair loss requiring a wig. After 1 year of treatment the tumor remains shrunken.

Drug B is taken in tablet form at home once a day. The drug is successful in shrinking the tumor. Out of 100 people who take the drug, 10 will experience occasional nausea which is usually relieved if taken with milk. Twenty will have occasional hot flushes which can be embarrassing on social occasions. After 1 year of treatment, the tumor remains shrunken.

ANNETTE M.C.

814

O'CONNOR

they were willing to make to avoid chemotherapy toxicity varied but were not related to diagnostic group or sex. The design did not control for experience with hormone therapy (mild nausea, hot flushes). It was assumed that most individuals would have experienced mild nausea at one time or other and hot flushes from exercise, nervousness, alcohol or caffeine ingestion, pregnancy or menopause. In addition, the scenario descriptions were supplemented by definitions of each of the symptoms. The men did not differ from the women in the rank ordering of side effects. For both groups, hot flushes were judged to be the least important side effect and nausea and vomiting the most important. The interviewer (AO) was referred to potential subjects by their attending physicians. Subjects who consented to participate were interviewed for 20-30 minutes in offices at the outpatient department or at the bedside or quiet area in the hospital. The interview was supplemented with typewritten cards and the interviewer recorded the patients’ responses. Interviews took place prior to the first treatment and were repeated about 6 weeks following initiation of therapy when subjects had had at least two courses of therapy. Personal data about the patients’ age, sex, diagnosis, and treatment were obtained. Linear analogue self assessment scales were used to measure perceived health and symptom severity. For example, patients rated the nausea they had experienced during the course of therapy by placing a mark on a 100 mm line which was anchored at one end by “no nausea” and at the other end by “extremely severe nausea”. These scales were previously tested for reliability and validity with similar patients [17] and found to be adequate. Data analysis focused on agreement and differences in utilities among elicitation techniques before treatment and change in utilities after treatment. Agreement among utilities was quantified using Pearson Y correlation coefficients. Differences in utilities by elicitation technique and on retest were determined using the paired t test. Correlations between utilities pre-treatment and post-initiation of treatment were also quantified using Pearson r correlation coefficients. RESULTS

The sample was obtained over a 6 month period. There were 65 patients who were avail-

etnl.

able to be interviewed during that time period. Four were not entered into the study due to severe depression (l), psychosis (l), low intelligence (1) and high doses of narcotics (1). An additional five patients received treatment before they could be interviewed; these patients did not differ in any discernable way (treatment, age, sex, diagnosis) from the data-producing sample. Fifty-six cancer patients were interviewed prior to treatment. Most interviews (85%) took place the day before or day of treatment and 98% took place within the week before treatment. The time lapse between the decision to have treatment and initiation of treatment ranged from one day to 3 weeks. Most patients had some notion that chemotherapy was a possible treatment for at least a week. Six of the 56 patients were unable to have a second interview due to deteriorating health or death and one refused a second course of treatment and a second interview. The patients were being treated with combination or single chemotherapeutic agents for breast (41 Oh), lung (41%), gastrointestinal (13%), or other (5%) types of cancer. The majority of subjects were female (56%) and the mean age of subjects was 54 with a range from 16 to 73 years. The frequency distributions of values for chemotherapy remission, obtained in the first interview using each elicitation technique, are presented in Fig. 1. Generally, values were very high and standard deviations were small; a notable exception was category rating where the mean was lower and the standard deviation wider. The mean differences among approaches is presented in Table 4. Differences were found between the riskless category rating and risky treatment choice approaches. No statistically significant differences were found between (a) the treatment choice questions when side effects were certain and uncertain, and (b) the Treatment Choice 100 and Treatment Choice 60 questions. The correlation coefficients among the pretreatment utilities for chemotherapy remission using the various elicitation techniques are presented in Table 5. There was good to excellent agreement among all the elicitation approaches except for category rating which was not correlated with any of the other approaches. The changes in symptoms before and during treatment are presented in Table 6. Patients reported significantly (p < 0.05) more nausea,

Eliciting Preferences

I

Side Effects

i:

0

0

7P1R

IO 20 Category rating

815 I

Certain

0.96

Side Effects

Uncertain

x=0.97

SD:0.09 Mdns0.99

SD= 0.06 Mdn: I.0

IO 20 Treotment choice 100

-7-z Treatment choice 60

0

IO 20 Treatment choice I00

0

IO 30 Treatment choice 60

Fig. 1.Frequency distribution, medians (Mdn), means @), and standard deviations (SD) of utilities/values for chemotherapy remission, obtained before treatment, using each elicitation technique, for certain and for uncertain side effects. The abscissa represents the frequency.

vomiting, fatigue, and hair loss after treatment. However, as indicated in Table 7, there were no significant differences in values for chemotherapy remission using any of the elicitation approaches. For all but category rating, the correlation coefficients between the utilities

before and during treatment were fair (0.4% 0.59). For the riskless category rating technique, where the correlation was poor (0.17), there were many changes in values; 68% of the scores increased, only 2% stayed the same and 30% went down. In contrast, the treatment choice

Table 4. Mean differences in utilities between elicitation techniques (N =51) jf diff.

f

1. Riskless Category Rating Risky Treatment Choice 100

-0.14

4.71

2. Side Effects Certain Treatment Choice 100 Treatment Choice 60

-0.01

-1.91

0.06

3. Side Effects Uncertain Treatment Choice 100 Treatment Choice 60

-0.004

-0.87

0.38

4. Treatment Choice 100 (side effects certain) Treatment Choice 100 (side effects uncertain)

-0.007

-1.61

0.11

Paired comparison

p value 0.001

Table 5. Correlation among utilities for chemotherapy remission obtained by various elicitation techniques Side effects certain

A. Side effects 1. Category 2. Treatment 3. Treatment

certain Rating Choice 100 Choice 60

B. Side effects uncertain 1. Treatment Choice 100 2. Treatment Choice 60

B. Side effects uncertain

2 Treatment Choice 100

3 Treatment Choice 60

1 Treatment Choice 100

0.08

0.04 0.87

0.06 0.95 0.83

2 Treatment Choice 60 0.001 0.84 0.87 0.80

816

ANNETTE M. C. O’CONNOR et al.

Table 6. Changes in symptoms before and 6 weeks after treatment commence@ Mean before treatment

Variable Perception of healthb Nausea’ Vomiting’ Fatigue’ Hair loss’ Hot flushes’ Pain’ Overall feeling of wellnessd

Mean after treatment

62 11 5 43 3 10 25 66

56 34, 26+ 62* 50* 20 21 62

‘Symptom severity was measured using 1OOmm linear analogue self assessment scales (see Methods). bO= worst possible health, 100 = best possible health. ‘0 = no side effect, 100 = extremely severe side effect. dO= overall feel terrible, 100 = overall feel well. *Post treatment score significantly different from pretreatment score, p = O.ooOl.

values increased for 19% of subjects, stayed the same for 58% and decreased for 23% of the patients. DISCUSSION

The major finding in this study was the lack of differences in utilities or preferences before and 6 weeks following initiation of treatment. The stability of patients’ utilities and preferences during the occurrence of the anticipated changes in symptoms is similar to the results of Llewellyn-Thomas et al. [9]. Just as values remained unchanged in a cancer context for a narrow range of functioning (voice quality), so too did values/utilities remain stable for a cancer treatment which affected a wider range of functioning (nausea, vomiting, fatigue, hair loss). The question which arises with this evidence is why the techniques can discriminate between groups differing in their health states [7,8], but do not detect changes as a function of time for the same individuals when changes in health state occur. Possible explanations are: (1) scaling problems; (2) the expected side effects were not severe enough, or (3) change may not occur until a later phase in treatment.

A scaling problem is the least likely explanation Although the risky treatment choice approach produced very skewed results, the riskless rating technique produced a better distribution which still did not detect a change. The risky elicitation procedure has been found in a current study [18] to discriminate between and within groups. The mean utility for chemotherapy remission for a group of chemotherapy patients at later phases of treatment (x = 0.92) differed significantly from the new chemotherapy patients in this study (x = 0.96). The utilities of the chemotherapy patients also differed significantly from radiotherapy patients (X = 0.90) healthy volunteers (x = 0.80) and nursing students (X = 0.79). Since this present study was conducted, within subject changes have also been detected when the chance of survival was lower than 60% [18]. The second possibility is that the changes in health states for the chemotherapy patients were not of sufficient magnitude to produce a change in values. The patients expected these symptoms and had agreed to the treatment to improve their chances of remission. Perhaps the change in health state was perceived to’ be low in magnitude when compared, for example, with the change from healthy to unhealthy with cancer. The low magnitude of effect is supported by the fact that 68% of subjects using the rating technique increased the value they assigned to chemotherapy remission after treatment commenced; only 30% using rating and 23% using the treatment choice method decreased the utilities assigned to chemotherapy remission after treatment commenced. A third plausible explanation is that subjects were interviewed too early in the treatment phase to detect changes. This conjecture is supported by the difference in utilities for those in the present study and those who were on treatment for a much later period of time [18]. Longitudinal studies of patients who receive high and low toxicity treatments need to be conducted.

Table 7. Changes and correlations in mean utilities/values Mean before treatment

Mean after treatment

Correlation

effects certain Category Rating Treatment Choice 100 Treatment Choice 60

0.82 0.96 0.97

0.86 0.96 0.97

0.17 0.54 0.59

Side effects uncertain 1. Treatment Choice 100 2. Treatment Choice 60

0.96 0.97

0.97 0.97

0.58 0.48

Side 1. 2. 3.

R

Eliciting Preferences

In this study, patients’ preferences were insensitive to the way information about side effects was presented (certain vs uncertain) and the level of probability that was presented (high vs moderate). This result indicates that preferences elicited from 60 to 100% survival will not be influenced by the level of probability. These results have been confirmed in a subsequent recent study [18]. The lack of sensitivity to treatment outcome descriptions indicates that a simpler format (certain approach) can be without substantially changing presented responses. The risky and non-risky techniques produced statistically significant differences in preferences. The values for chemotherapy remission using rating were different from the utilities assessed by the treatment choice approaches, probably because subjects were not risk neutral. Subjects were considered risk averse because the mean value for chemotherapy remission using rating was much lower than the mean utility obtained from the treatment choice questions. The poor correlation between the techniques indicates that those who thought a chemotherapy remission was quite undesirable were not necessarily those who were willing to trade off probability of survival to avoid toxic side effects. Indeed, the patients’ extreme reluctance to trade off survival before and six weeks into treatment is remarkable. The median patient in the sample preferred a less toxic treatment over a more toxic treatment when the chances of survival were both the same, but was unwilling to trade off survival rate to reduce the chance of side effects. The data appeared to support the decision for chemotherapy which was made prior to the subjects’ entry into the study. An alternative explanation may be that subjects made their preferences consonant with the decision that was made and remained consonant even after experiencing toxicity. Until preferences are elicited prior to treatment choice, the question of whether preferences truly represent the patient’s point of view or are the result of cognitive dissonance reduction will remain unanswered. Questions about the extent to which patients are willing, at the time of decision making, to trade off survival rate for improved quality of life should also be raised. The chemotherapy group has had the highest reported utilities and narrowest standard deviations compared to other studies [2,8, 16, 181 where patients were interviewed at a later phase in treatment. These

817

differences may be attributed to differences in patient characteristics, but may also be due to the treatment phase. Perhaps the extreme risk aversion manifested around the time of decision making may not be as evident at a later time. If this were the case, then the assertion that quality of life should be considered in addition to quantity of life [19,20] may not be valid. It would also raise questions about the validity of health services research where quality adjusted survival is used as a basis for health care policy decisions. On the other hand, this interpretation is based on the assumption that the preferences at the time of decision making are the ones that count. One could argue that patients’ utilities at later phases of treatment should be considered if it were known that quality of life becomes more of an issue in subsequent phases. There is an obvious need to test this interpretation with a cohort of patients who have yet to make a decision about treatment. The convergent and discriminant validity of techniques should then be examined at several points during treatment beginning with the phase prior to choice (the deliberation phase). If it is found that some patients are willing to trade off probability of survival for improved quality of life and if the assessment techniques not only validate decisions made for a more toxic treatment as they did in this study but also validate decisions to reject or drop out of treatment, then clinical applications are evident. The techniques could be used prior to decision making to screen for individuals whose preferences appear inconsistent or intransitive. This group could be referred for further counselling to clarify the information presented and help patients articulate their values and preferences. The techniques could also be used on an ongoing basis to monitor acceptance of treatment. Changes in values could be explored with patients and other treatment alternatives examined if the need arises. The techniques may not only be useful when patients are actively involved in decision making about alternative treatments but also in the case where clinicians obtain informed consent from patients for risky procedures or treatments. Consistency among responses may indicate that patients truly understand or accept the implications of the treatment or the procedure they will undergo. Finally, the information about possible long term and short term changes in preferences, values, and behaviour over time will be useful to patients and families in understanding the

ANNETTEM. C. O’CONNORet al.

818

impact that treatment has had on others like themselves. In summary, the results of this study appeared to support the decision about treatment that had been made; subjects expressed an overwhelming preference for chemotherapy. With the exception of category rating, there was excellent agreement among elicitation techniques. The patients values for treatment outcomes remained stable despite the toxicity they experienced. Their extreme reluctance to make tradeoffs, compared to other studies, questions the extent to which patients close to the time of decision making are willing to trade off survival rate for better quality of life. The elicitation techniques may have useful clinical applications once validation studies with patients who are actually facing treatment decisions are completed. Acknowledgements-A.M.C. O’C. held a Connaught Fellowship. This work was supported in part by the Ontario Cancer Treatment and Research Foundation and the National Cancer Institute of Canada. The authors are grateful to the patients who participated and the medical and nursing staff at the Ottawa Clinic of the Ontario Cancer Foundation.

10.

11.

12. 13.

14.

15.

REFERENCES 1. O’Connor AM, Boyd NF, Jill JE: Methodological problems in assessing preferences for alternative therapies in oncology: the influence of preference elicitation technique, position order, and the test-retest error on preferences for alternative cancer drug therapies. Proc 10th NationaI Nursing Research Conference. University of Toronto, 1986. pp. 49-58 2. Llewellyn-Thomas H, Sutherland HJ, Tibshirani R, Ciampi A, Till JE, Boyd NF: Describing health states: Methodologica issues in obtaining values for health states. Med Care 22: 543-552, 1984 3. Torrance GW: Social preferences for health states: An empirical evaluation of three measurement techniques. !Sacio+con PIan Sci 10: 129-136, 1976 4. Kahneman D, Tversky A: Prospect theory: An analysis of decision under risk. EronomeMcr 47: 263-29 1,1979 5. Hershey JC, Kunreuther HC, Schoemaker PJH: Sources of bias in assessment procedures for utility functions. Management Sci 28: 936-954, 1982

16.

17.

18.

19.

20.

Christensen-Szalanski JJJ: Discount functions and the measurement of patients’ values: Women’s decisions during childbirth. Med DeeIs Making 4: 47-58, 1984 Sackett DL, Torrance GW: The utility of different health states as perceived by the general public. J Chron Dis 31: 697-704, 1978 Boyd NF, Sutherland HJ, Cummings BJ: The selection of primary therapy for patients with cancer of the rectum. Med De& Making 2: 354 1982 (Abstract) Llewellyn-Thomas HA, Sutherland HJ, Ciampi A, Etezadi-Amoli J, Boyd NF, Till JE: The assessment of values in laryngeal cancer: Reliability of measurement methods. J Cluon Dis 37: 283-291, 1984 Fischhoff B, Slavic P, Lichtenstein S: Knowing what you want: Measuring labile values. Cognitive Processes in Choice and Decision Behavior, Wallsten TS (Ed.). Hillsdale, N.J.: Erlbaum, 1980. pp. 117-141 Bonadonna G, Rossi A, Valagussa P, Banfi A, Veronesi U: The CMF program for operable breast cancer with positive axillary nodes. Cancer 39: 29042915, 1977 Haskell CM: Systemic therapy for metastic breast cancer. AM Intern Med 86: 68-80, 1977 Ingle JN, Ahmann DL, Green SJ, Edmonson JH, Bisel HF, Kvols LK, Nichols WC, Creagan ET, Hahn RG, Rubin J, Frytak S: Randomized clinical trial of Diethylstilbestrol versus Tamoxifen in postmenopausal women with advanced breast cancer. N Engl J Med 304: 16-21, 1981 Legha SS, Davis HL, Muggia FM: Hormonal therapy of breast cancer: New approaches and concepts. Ann Intern Med 88: 69-77, 1978 Cavalli F, Beer M, Martz G, Jungi WF, Albcrto P, Obrecht JP, Mermillod B, Brunner KW. Concurrent or sequential use of cytotoxic chemotherapy and hormone treatment in advanced breast cancer: Report of the Swiss Group for Clinical Cancer Research. Br Med J 286: 5-8, 1983 Llewellyn-Thomas H, Sutherland HJ, Tibshirani R, Ciamoi A. Till JE. Bovd NF: The measurement of patients’ values in’ medicine. Med Decis Making 2: 449462, 1982 Selby PJ, Chapman JAW, Etezadi-Amoli J, Dalley D, Boyd NF: The development of a method for assessing the quality of life of cancer patients. Br J Cancer 50: 13-22, 1984 O’Connor A, Thiel E, Durham L, Joanisse S: Explaining risks to patients: The influences of framing and level of probability on preferences. Proe Int Nursing Research Conference. Edmonton, Alberta: University of Alberta. In press McNeil BJ, Weichselbaum R, Pauker SG: Fallacy of the five-year survival in lung cancer. N Engl J Med 299: 1397-1401, 1978 McNeil BJ, Weichselbaum R, Pauker SG: Speech and survival: Tradeoffs between quality and quantity of life in laryngeal cancer. N Engl J Med 305: 982-987, 1981