- Email: [email protected]
Using Multicriteria Approaches to Assess the Value of Health Care
VALUE IN HEALTH 20 (2017) 251–255
Available online at www.sciencedirect.com
journal homepage: www.elsevier.com/locate/jval
Using Multicriteria Approaches to Assess the Value of Health Care Charles E. Phelps, PhD1,*, Guruprasad Madhavan, PhD2 1
University of Rochester, Rochester, NY, USA; 2The National Academies of Sciences, Engineering and Medicine, Washington, DC, USA
AB STR A CT
Practitioners of cost-utility analysis know that their models omit several important factors that often affect real-world decisions about health care options. Furthermore, cost-utility analyses typically reflect only single perspectives (e.g., individual, business, and societal), further limiting the value for those with different perspectives (patients, providers, payers, producers, and planners—the 5Ps). We discuss how models based on multicriteria analyses, which look at problems from many perspectives, can fill this void. Each of the 5Ps can use multicriteria analyses in different ways to aid their decisions. Each perspective may lead to different value measures and outcomes, whereas no single-metric approach (such as cost-utility analysis) can satisfy all these stakeholders. All stakeholders have unique ways to measure value, even if assessing the same health intervention. We illustrate the benefits of this approach by comparing the value of five different hypothetical treatment choices for five hypothetical patients
with cancer, each with different preference structures. Nine attributes describe each treatment option. We add a brief discussion regarding the use of these approaches in group-based decisions. We urge that methods to value health interventions embrace the multicriteria approaches that we discuss, because these approaches 1) increase transparency about the decision process, 2) allow flight simulatortype evaluation of alternative interventions before actual investment or deployment, 3) help focus efforts to improve data in an efficient manner, 4) at least in some cases help facilitate decision convergence among stakeholders with differing perspectives, and 5) help avoid potential cognitive errors known to impair intuitive judgments. Keywords: multicriteria analysis, priority setting, systems analysis, value modeling.
Introduction
approach using only a single attribute and a single perspective is almost certainly inadequate and incomplete. Others have acknowledged this challenge. For example, the International Society for Pharmaceutical Outcomes Research (ISPOR) Task Force on multicriteria decision analysis issued two reports in 2016 summarizing best practices for such approaches [4,5]. These reports—in addition to providing a state-of-the-art review—discuss the uses of the multicriteria models in health care, from supporting decisions of individual patients to a universal health care system. As their summaries show, use of multicriteria analysis in the United States has primarily focused on decision support for individual patients rather than on technology assessment or insurance coverage decisions (e.g., [6]). Among the 5Ps, Patients are likely to focus on potential therapeutic benefits, risks, and side-effects, as well as travel time, net costs and out of pocket expense as they evaluate treatment options and other alternatives, including watchful waiting, palliation, treatment holidays, and continuous monitoring. Providers may consider their understanding of patient preferences, but may also consider the consequences for their own financial situation (including differential reimbursement
Standard approaches for evaluating health interventions use cost-utility analysis, most commonly with a societal perspective. All health-improving benefits and costs enter the cost-utility model (similar to the methods of cost-benefit analysis), which follows directly from a utility-maximizing framework [1,2]. Costeffectiveness and cost-benefit analyses are closely related once a critical cutoff value is established for resource allocation in the cost-utility framework [3]. But this perspective seldom corresponds to the perspectives of most of the participants in the health enterprise: patients, providers, payers, producers, and planners—the 5Ps. Each of these 5Ps almost certainly will have multiple objectives when selecting, advising about, providing coverage for, investing in, or supporting research about health care options. Each of these decision makers has multiple goals, and therefore no single-metric approach (such as cost-benefit or cost-utility ratios) can accommodate any of their viewpoints. We need better, comprehensive frameworks to assess the value of health that can both accommodate different values and incorporate multiple attributes associated with health options. Any
Copyright & 2017, International Society for Pharmacoeconomics and Outcomes Research (ISPOR). Published by Elsevier Inc.
Note: The views expressed in this article are those of the authors and not necessarily of the National Academies of Sciences, Engineering, and Medicine. * Address correspondence to: Charles E. Phelps, 30250 South Highway 1, Gualala, CA 95445. E-mail: [email protected]. 1098-3015$36.00 – see front matter Copyright & 2017, International Society for Pharmacoeconomics and Outcomes Research (ISPOR). Published by Elsevier Inc. http://dx.doi.org/10.1016/j.jval.2016.11.011
252
VALUE IN HEALTH 20 (2017) 251–255
and use of provider-owned facilities), and perhaps litigation risks. Payers (third-party insurers) may seek to minimize premium costs by bargaining with providers, perhaps choosing limited panels of preferred providers, and choosing which therapies to include as covered and at what co-payment rates (e.g., tiers within prescription drug plans). Producers (e.g., manufacturers of drugs, vaccines, and devices) may each have their own interests, ranging from profitability to perceived public good. Planners may focus on prioritizing research investments and possibly (for elected policymakers) even on enhanced re-election chances. Again, no single perspective can possibly represent these diverse interests.
cognitive biases include those relating to framing of issues, anchoring, time inconsistency, estimation of probabilities, making choices in the presence of uncertainty, overconfidence, and many others [15]. These five considerations—transparency, advance testing of alternatives, efficiency in data improvement, supporting decision convergence, and alleviation of cognitive biases in real-world decision making—all point to the value of using multicriteria models. Most previous assessments of multicriteria models have primarily emphasized the value of transparency in the decision process, although we believe that the other considerations listed here create similar benefits.
Why Use Multifactorial Systems Analysis?
Present Limitations of Multicriteria Approaches
Multifactorial approaches approximate as well as expand the economic concept of a utility function with a goal of incorporating relevant parameters to make the models useful for decision support [7–11]. Each approach makes trade-offs between generalizability and usability. They also seek to consider competing attributes of various options, making the trade-offs explicit and transparent to others, thus excelling in comparison with singleattribute models such as cost-effectiveness analysis [12]. Multicriteria approaches may lead to two potentially conflicting situations. First, all users can create their own priority lists on the basis of their own values and preferences. Second, a consequence of the first, we may see the creation of potentially disparate lists of priorities. This is not a defect of multicriteria models, but rather represents the reality of the world: people with different perspectives have different priorities. Multicriteria approaches provide a number of direct benefits related to decision making. First comes transparency. These models illuminate the key values driving priorities and choices [13]. Second, this approach allows the assessment of the implications of adopting a particular option (and multiple variations thereof) before actually spending resources to implement the choice, similar to a flight simulator in aviation engineering. This advance testing capability uniquely builds on the multicriteria evaluation processes of these models. In appropriate settings, it can also allow reverse engineering to improve the specifications or characteristics of diagnostic or treatment interventions (e.g., considering trade-offs between cost, discomfort, radiation exposure, and accuracy in diagnostic protocols). Third, multicriteria models can help guide efforts to systematically improve data. Often, at least some of the data entering the model are imperfect. Using sensitivity analysis within the model can show which variables importantly contribute to the final rankings and which do not matter, allowing users to focus their efforts on improving those data that matter most [14]. Fourth, multicriteria models can potentially assist competing interests in reaching negotiated decision convergence. Prices in markets perform a similar function by providing information to buyers and sellers about how other participants in the market value things. Similarly, values specified in multicriteria models provide information that can assist various parties in negotiating agreements by showing what is more important and less important to various stakeholders. Clearly, in some settings, various bargaining parties may not wish to reveal such information, but in some other settings, it can pave the way to a mutually agreeable negotiated agreement, just as market prices help to facilitate exchanges. Fifth, multicriteria approaches can help bypass at least some cognitive biases that are now well understood to adversely affect decisions. These models ask humans to do tasks for which they are best suited (specifying values) and then ask computers to do what they do best (numerical synthesis of information). These
A key limitation of multicriteria models is the lack of consensus on the best method to balance costs and benefits because benefit measures in these models purposefully include a wider array of attributes than do standard narrow measures such as qualityadjusted life-years (QALYs). A recent ISPOR Task Force report discussed three ways to approach these issues using multicriteria analysis [5]. The first uses direct inclusion of cost as a “negative” attribute with its own weight. The second involves finding other health care interventions (with known costs) that could be eliminated and assessing the multicriteria value of those as a benchmark. The third approach divides the multicriteria scores of each option by its cost (akin to using the ratio of cost per health benefit). The task force found none of these approaches perfectly persuasive, and urge further research on the matter. A fourth available alternative assumes an exogenous investment budget, without suggesting how to set its level [16]. We suggest a new alternative here. Cost-benefit analysis recommends adopting every alternative choice with positive net present value. This method, however, requires analysts to assign a specific value to human life, a task many resist. To solve this concern, cost-utility analysis calculates the incremental cost-effectiveness ratio (ICER), and the decision maker chooses the critical cutoff value for acceptable ICERs. We suggest a similar approach for use in multicriteria models. Suppose that in any multicriteria model, health-related measures (such as QALYs) accounted for a share S of the total value, and all other attributes accounted for a share of 1 S. Then, if the ICER cutoff was $100,000/QALY, the measured scores for each choice in the multicriteria model would be evaluated against a cutoff of $100,000/S. This would provide a direct method, comparable with that of cost-utility analysis when using an ICER cutoff to guide resource allocation in the broader framework of multicriteria decision analysis. Other concerns about multicriteria models are specific to particular applications. For example, the weight-setting protocol in the analytic hierarchy process allows internal inconsistencies and rank-reversal with different decision options [11]. The swing weight approach in the multi-attribute utility theory requires analysts to create standardized 0 to 100 scales for each attribute, which interact with the weights in potentially unforeseen ways [9]. These and other technical issues can and should be resolved to bring multicriteria models to their greatest potential.
Choices and Scenarios To illustrate key features of the multicriteria systems approach, consider a patient faced with choosing among available cancer therapies. Clinicians would specify the quantitatively measurable therapeutic and side-effect profiles of each option. Patients would specify the levels of any subjectively determined attribute associated with each choice, for example, distaste or fear of a
253
(0.148) (0.026) (0.012) (0.061) (0.012) 3 3 9 6 9 (0.026) (0.042) (0.314) (0.042) (0.148) 2 5 8 8 7
(0.203) (0.083) (0.026) (0.026) (0.042)
8 7 1 7 3
0 0 1 0 0 5,000 5,000 0 2,000 1,000
(0.314) (0.012) (0.111) (0.314) (0.061) 1 9 4 1 6 A B C D E
6 1 2 2 1
(0.061) (0.314) (0.203) (0.203) (0.314)
4 4 7 3 8
A. Treatment performance on nine attribute scales 0.8 0.3 2 50,000 0.2 0.8 8 50,000 0.3 0.4 5 80,000 0.8 0.7 3 20,000 0 0 10 2,000 B. Patient rankings of attributes (and associated weights) (0.111) 5 (0.083) 7 (0.042) 9 (0.012) (0.111) 3 (0.148) 2 (0.203) 6 (0.061) (0.042) 6 (0.061) 5 (0.083) 3 (0.148) (0.148) 4 (0.111) 5 (0.083) 9 (0.012) (0.026) 5 (0.083) 4 (0.111) 2 (0.203) 12 24 12 36 0
Probability of nausea (0–1.0) Probability of hair loss (0–1.0) Expected months of remission-free survival (0–36) Probability of remission (0– 0.5)
0.4 0.2 0.2 0.1 0
We also created five hypothetical patients with the same primary cancer who have different preference profiles and attribute ranks. The weights associated with these rankings come from the
T1 T2 T3 T4 T5
Patients
Treatment/ patient
Multi-attribute utility models require defining boundaries of the worst and best outcomes that can be imagined for the choices. Thus, for example, we set the best case for probability of remission at 0.5 (rather than 1.0), under the premise that for this disease, higher probabilities of remission cannot be imagined with existing treatment approaches. Table 1A reports the values of each of these five hypothetical therapies (T1–T5) on each of their nine attributes, showing (in the column headings) the feasible range and (where appropriate) which score is the best. In our example, all these attributes are quantified, whereas in other settings, some of the attributes might be subjectively determined, as previously discussed. T1 has the highest probability of remission, but lower overall survival if the regimen fails. It is supported by a moderate-sized randomized controlled trial (RCT), has mid-range costs to patient and society, and is quite likely to cause pain, hair loss, and considerable nausea. T2 is similar to T1, with a lower remission rate but potentially longer overall survival. It has stronger RCT evidence with higher expected months of remission-free survival. Costs are the same. It has less pain associated with it, but a higher chance of nausea. T3 is offered in an RCT. Preliminary evidence says it has high survival benefits and life extension, but is available to the patient only with 50% probability, and so the probabilities of remission and life extension reflect the expected value of the treatment and the control therapy in the RCT, and the side-effect profiles are a similar mix of the experimental and control therapies. T3 has a higher societal cost, but is free to the patient as part of the RCT. T4 has a low remission rate, but the highest expected remission-free survival. It is opposite of T1. It gives the most assurance for some additional survival, but little promise of remission. It has relatively low cost, and has strong RCT evidence supporting its outcomes. It has a side-effect profile that is unfavorable relative to other options. T5 is palliative care only. It offers no remission or life expectancy gains, but has no side effects and removes all pain associated either with therapy or with the disease itself. It costs little, and the data supporting its outcomes are solid.
Table 1 – Treatment performance and patient preferences based on multiple attributes.
Treatments
Pain scale (0–10)
Total cost ($1K–$100K)
Patient cost ($0–$10K)
Advancing knowledge (0 or 1)
Quality of evidence about benefits (0–5)
particular therapeutic modality (e.g., radiation, chemotherapy, or molecular targeting) independent of quantitatively known benefits and risks. As a specific example, we show the choice of five hypothetical cancer therapies for a set of hypothetical individuals using multiattribute utility theory, a specific approach within multicriteria systems analysis. This approach requires the decision maker (patient) to specify weights for the selected attributes. A simple method to obtain good weights simply asks the patient to list the desired attributes in rank order. Then, a formula—the rank order centroid method —estimates the average of all weights (summing to 1.0) consistent with the selected rankings [9]. This approximation has been shown to provide highly reliable results when used in linear multi-attribute models [10]. Our example uses rank order lists from the hypothetical patients to rank the importance of nine attributes in cancer therapy, which then create corresponding weights. The multi-attribute model combines the attribute performance measures of each therapy with the patients’ weights to create value scores on a 0 to 100 scale.
4 5 2 5 4
VALUE IN HEALTH 20 (2017) 251–255
254
VALUE IN HEALTH 20 (2017) 251–255
previously discussed rank order centroid method [9,10]. Table 1B summarizes each of these hypothetical patient’s preferences, showing the rank order in which they place each of the nine attributes of the therapies and (in parentheses) the associated rank order centroid weights. In short, our five patients have the following value structures. Patient A places the greatest value on the chance of a remission, and then out-of-pocket cost, and wants certainty about the quality of the evidence, followed by concerns about hair loss and nausea, freedom from pain, advancement of knowledge, and last, total societal cost. Patient B places the greatest value on expected months of remission-free survival, but has little concern about the probability of remission. This patient has heightened concerns about treatment side effects, modest concern for out-ofpocket costs and total costs, and little concern about advancing knowledge or the quality of data supporting the doctors’ recommendations. Patient C has a different view, committed to advancing societal knowledge, concerned about the total cost of care, but wishing to avoid side effects if possible. This patient has moderate interest in the probability of remission, and high interest in extending life expectancy, wishing primarily to survive until a granddaughter’s planned wedding. Patient D is self-oriented, caring only about survival, remission, pain, and side effects, and little at all about social or private cost, advancing knowledge, or the quality of the data. Patient E cares considerably about extending life expectancy, but is guilt-ridden about using resources that do not lead to remission, and so has great concern about the total cost of treatment, and also wishes to contribute to human knowledge about curing this disease. Modestly concerned about side effects, this patient has full insurance coverage and hence minimal concerns about out-of-pocket costs. Table 2 highlights each hypothetical patient’s most preferred choice of therapy. In some cases (e.g., patient A), the choices are close in value, whereas in other cases (e.g., patient B) they differ considerably. These scenarios illustrate our key point: preferences matter. Each of these hypothetical patient profiles could plausibly represent a real person and their trade-off considerations. Although these profiles were constructed deliberately to make our point, clearly the creation of preference weights that lead to different choices is not difficult. Indeed, our experience with using multi-attribute models in the realm of global health for prioritizing new vaccine development has led us to expect that differences in preference structure commonly lead to different outcomes in ranking of alternatives [17–19]. The values in Table 2 call to attention another feature of multi-attribute models: The scores of the different patients are unrelated to each other. Each patient’s preference structure creates a different value scale, just as, for example, Fahrenheit and Celsius temperature scales differ. Furthermore, these scales are not relative values. A value of 60 is not twice as good as a
Table 2 – Treatment scores based on patient values. Patient A B C D E
T1
T2
T3
T4
T5
58.6 37.4 32.0 51.1 37.1
56.0 59.3 39.1 54.8 49.5
58.5 52.5 59.4 50.0 48.8
50.3 58.5 42.8 44.5 59.8
54.9 61.6 36.4 42.5 46.7
Note. Figures in boldface represent the highest ranked treatment for each patient.
value of 30; it is 30 points better, just as 801F is not twice as warm as 401F, but is 401 warmer. These are interval, but not ratio, scales.
Choosing by Groups Although our hypothetical example has a single decision maker, the same conceptual approach remains valid in group decisionmaking processes. Groups would use one of several available voting methods to rank attributes in the model and to select (when necessary) appropriate values for any subjectively defined attributes. Each of these requires a collective choice, using a voting or grading process of some form, guidance for which to use is found in the social choice literature [20]. When might such issues appear? In health care, the occasions may involve investment choices, personnel decisions and related provision of care issues, or perhaps insurance coverage decisions. Suppose a prepaid managed care plan or an accountable care organization wished to choose the best cancer therapy for a particular condition. It might use samples of real patients to help make a system-wide choice among therapeutic options. Many other resource planning decisions have the same basic structure. The recent ISPOR Task Force reports provide useful examples of these types of multicriteria applications in group settings [4,5]. On the basic research end of the spectrum, review panels of governments or charitable foundations evaluate various grant proposals geared toward improving cancer therapy. Often in the current processes, peer reviewers give a composite numerical score to proposals under their consideration, with scores averaged to achieve final ranking recommendations. As an alternative, they could evaluate each proposal on the basis of various research attributes—uniqueness, likelihood of scientific breakthrough, public impact—and then use multicriteria models to provide final scoring of research proposals.
Conclusions Once various technical issues are resolved, we believe that multicriteria systems analysis models have much to favor them over alternative approaches used to guide health care choices, and possibly in broader program evaluation and comparisons. For decisions ranging from individual to societal, multicriteria models provide a valuable extension over single-criterion models such as cost-utility analysis. The participants in the health care enterprise (i.e., the 5Ps) commonly hold different perspectives, and so no single perspective can possibly meet all their needs. Using systems-based approaches improves transparency about their decision processes, because all assumptions and values are made specific. Often, these models can guide improvements in product design by varying attributes and costs of alternatives and measuring the value created by such variations. At least in some settings, this transparency can facilitate decision convergence among otherwise competing viewpoints. These models also help focus efforts to improve data in an efficient manner using sensitivity analysis. Finally, using these approaches also reduces risks of cognitive error associated with intuitive judgments and choices. In our view, these gains in assisting decision making at various levels provide compelling reasons to shift from narrow analytic techniques to systems approaches capable of incorporating a wide array of factors that actually drive real decisions in health and medicine.
VALUE IN HEALTH 20 (2017) 251–255
R EF E R EN CE S
[1] Garber A, Phelps C. Economic foundations of cost-effectiveness analysis. J Health Econ 1997;16:1–31. [2] Neumann PJ, Sanders GD, Russell LD, et al., editors. Cost-Effectiveness in Health and Medicine (2nd ed.). New York, NY: Oxford University Press, 2017. [3] Phelps C, Mushlin A. On the (near) equivalence of cost-effectiveness and cost-benefit analyses. Int J Technol Assess Health Care 1991;7:12–21. [4] Thokala P, Devlin N, Marsh K, et al. Multiple criteria decision analysis for health care decision making—an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health 2016;19:1–13. [5] Marsh K, IJzerman M, Thokala P, et al. Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health 2016;19:125–37. [6] Dolan J. Multi-criteria clinical decision support: a primer on the use of multiple criteria decision making methods to promote evidence based, patient-centered healthcare. Patient 2010;3:229–48. [7] Keeney RL, Raiffa H. Decisions with Multiple Objectives. Cambridge, UK: Cambridge University Press, 1993. [8] Keeney RL. Value Focused Thinking: A Path to Creative Decision Making. Cambridge, MA: Harvard University Press, 1992. [9] Edwards W, Barron FH. SMARTS and SMARTER—improved simple methods for multiattribute utility measurement. Organ Behav Hum Decis Process 1994;60:306–25.
255
[10] Barron FH, Barrett BE. Decision quality using ranked attribute weights. Manag Sci 1996;42:1515–23. [11] Saaty TL. Decision Making for Leaders: The Analytic Hierarchy Process for Decisions in a Complex World. Pittsburgh, PA: RWS Publications, 2008. [12] Phelps CE, Madhavan G, Rappuoli R, et al. Beyond cost-effectiveness: using systems analysis for infectious disease preparedness. Vaccine. In press. [13] Madhavan G, Phelps CE, Colwell R, et al. In plain view: a transparent systems approach for enhancing health policy decisions. Bridge (National Academy of Engineering) 2016;42:5–10. [14] Phelps CE, Mushlin AI. Focusing medical technology assessment using medical decision theory. Med Decis Making 1988;8:279–89. [15] Kahneman D. Thinking, Fast and Slow. New York, NY: Farrar, Straus and Giroux, 2011. [16] Phillips LD, Bana e Costa CA. Transparent prioritization, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing. Working Paper LSEOR 05.75, London School of Economics, 2005. [17] Phelps C, Madhavan G, Sangha K, et al. A priority-setting aid for new vaccine candidates. Proc Natl Acad Sci 2014;111:3199–200. [18] Madhavan G, Sangha K, Phelps C, et al., editors. Ranking Vaccines: A Prioritization Software Tool. Washington, DC: The National Academies Press, 2013. [19] Madhavan G, Phelps C, Rappuoli R, et al., editors. Ranking Vaccines: Applications of a Prioritization Software Tool. Washington, DC: The National Academies Press, 2015. [20] Pacuit E. Voting methods. 2012. Available from: http://plato.stanford.edu/ archives/win2012/entries/voting-methods/. [Accessed December 17, 2016].
Available online at www.sciencedirect.com
journal homepage: www.elsevier.com/locate/jval
Using Multicriteria Approaches to Assess the Value of Health Care Charles E. Phelps, PhD1,*, Guruprasad Madhavan, PhD2 1
University of Rochester, Rochester, NY, USA; 2The National Academies of Sciences, Engineering and Medicine, Washington, DC, USA
AB STR A CT
Practitioners of cost-utility analysis know that their models omit several important factors that often affect real-world decisions about health care options. Furthermore, cost-utility analyses typically reflect only single perspectives (e.g., individual, business, and societal), further limiting the value for those with different perspectives (patients, providers, payers, producers, and planners—the 5Ps). We discuss how models based on multicriteria analyses, which look at problems from many perspectives, can fill this void. Each of the 5Ps can use multicriteria analyses in different ways to aid their decisions. Each perspective may lead to different value measures and outcomes, whereas no single-metric approach (such as cost-utility analysis) can satisfy all these stakeholders. All stakeholders have unique ways to measure value, even if assessing the same health intervention. We illustrate the benefits of this approach by comparing the value of five different hypothetical treatment choices for five hypothetical patients
with cancer, each with different preference structures. Nine attributes describe each treatment option. We add a brief discussion regarding the use of these approaches in group-based decisions. We urge that methods to value health interventions embrace the multicriteria approaches that we discuss, because these approaches 1) increase transparency about the decision process, 2) allow flight simulatortype evaluation of alternative interventions before actual investment or deployment, 3) help focus efforts to improve data in an efficient manner, 4) at least in some cases help facilitate decision convergence among stakeholders with differing perspectives, and 5) help avoid potential cognitive errors known to impair intuitive judgments. Keywords: multicriteria analysis, priority setting, systems analysis, value modeling.
Introduction
approach using only a single attribute and a single perspective is almost certainly inadequate and incomplete. Others have acknowledged this challenge. For example, the International Society for Pharmaceutical Outcomes Research (ISPOR) Task Force on multicriteria decision analysis issued two reports in 2016 summarizing best practices for such approaches [4,5]. These reports—in addition to providing a state-of-the-art review—discuss the uses of the multicriteria models in health care, from supporting decisions of individual patients to a universal health care system. As their summaries show, use of multicriteria analysis in the United States has primarily focused on decision support for individual patients rather than on technology assessment or insurance coverage decisions (e.g., [6]). Among the 5Ps, Patients are likely to focus on potential therapeutic benefits, risks, and side-effects, as well as travel time, net costs and out of pocket expense as they evaluate treatment options and other alternatives, including watchful waiting, palliation, treatment holidays, and continuous monitoring. Providers may consider their understanding of patient preferences, but may also consider the consequences for their own financial situation (including differential reimbursement
Standard approaches for evaluating health interventions use cost-utility analysis, most commonly with a societal perspective. All health-improving benefits and costs enter the cost-utility model (similar to the methods of cost-benefit analysis), which follows directly from a utility-maximizing framework [1,2]. Costeffectiveness and cost-benefit analyses are closely related once a critical cutoff value is established for resource allocation in the cost-utility framework [3]. But this perspective seldom corresponds to the perspectives of most of the participants in the health enterprise: patients, providers, payers, producers, and planners—the 5Ps. Each of these 5Ps almost certainly will have multiple objectives when selecting, advising about, providing coverage for, investing in, or supporting research about health care options. Each of these decision makers has multiple goals, and therefore no single-metric approach (such as cost-benefit or cost-utility ratios) can accommodate any of their viewpoints. We need better, comprehensive frameworks to assess the value of health that can both accommodate different values and incorporate multiple attributes associated with health options. Any
Copyright & 2017, International Society for Pharmacoeconomics and Outcomes Research (ISPOR). Published by Elsevier Inc.
Note: The views expressed in this article are those of the authors and not necessarily of the National Academies of Sciences, Engineering, and Medicine. * Address correspondence to: Charles E. Phelps, 30250 South Highway 1, Gualala, CA 95445. E-mail: [email protected]. 1098-3015$36.00 – see front matter Copyright & 2017, International Society for Pharmacoeconomics and Outcomes Research (ISPOR). Published by Elsevier Inc. http://dx.doi.org/10.1016/j.jval.2016.11.011
252
VALUE IN HEALTH 20 (2017) 251–255
and use of provider-owned facilities), and perhaps litigation risks. Payers (third-party insurers) may seek to minimize premium costs by bargaining with providers, perhaps choosing limited panels of preferred providers, and choosing which therapies to include as covered and at what co-payment rates (e.g., tiers within prescription drug plans). Producers (e.g., manufacturers of drugs, vaccines, and devices) may each have their own interests, ranging from profitability to perceived public good. Planners may focus on prioritizing research investments and possibly (for elected policymakers) even on enhanced re-election chances. Again, no single perspective can possibly represent these diverse interests.
cognitive biases include those relating to framing of issues, anchoring, time inconsistency, estimation of probabilities, making choices in the presence of uncertainty, overconfidence, and many others [15]. These five considerations—transparency, advance testing of alternatives, efficiency in data improvement, supporting decision convergence, and alleviation of cognitive biases in real-world decision making—all point to the value of using multicriteria models. Most previous assessments of multicriteria models have primarily emphasized the value of transparency in the decision process, although we believe that the other considerations listed here create similar benefits.
Why Use Multifactorial Systems Analysis?
Present Limitations of Multicriteria Approaches
Multifactorial approaches approximate as well as expand the economic concept of a utility function with a goal of incorporating relevant parameters to make the models useful for decision support [7–11]. Each approach makes trade-offs between generalizability and usability. They also seek to consider competing attributes of various options, making the trade-offs explicit and transparent to others, thus excelling in comparison with singleattribute models such as cost-effectiveness analysis [12]. Multicriteria approaches may lead to two potentially conflicting situations. First, all users can create their own priority lists on the basis of their own values and preferences. Second, a consequence of the first, we may see the creation of potentially disparate lists of priorities. This is not a defect of multicriteria models, but rather represents the reality of the world: people with different perspectives have different priorities. Multicriteria approaches provide a number of direct benefits related to decision making. First comes transparency. These models illuminate the key values driving priorities and choices [13]. Second, this approach allows the assessment of the implications of adopting a particular option (and multiple variations thereof) before actually spending resources to implement the choice, similar to a flight simulator in aviation engineering. This advance testing capability uniquely builds on the multicriteria evaluation processes of these models. In appropriate settings, it can also allow reverse engineering to improve the specifications or characteristics of diagnostic or treatment interventions (e.g., considering trade-offs between cost, discomfort, radiation exposure, and accuracy in diagnostic protocols). Third, multicriteria models can help guide efforts to systematically improve data. Often, at least some of the data entering the model are imperfect. Using sensitivity analysis within the model can show which variables importantly contribute to the final rankings and which do not matter, allowing users to focus their efforts on improving those data that matter most [14]. Fourth, multicriteria models can potentially assist competing interests in reaching negotiated decision convergence. Prices in markets perform a similar function by providing information to buyers and sellers about how other participants in the market value things. Similarly, values specified in multicriteria models provide information that can assist various parties in negotiating agreements by showing what is more important and less important to various stakeholders. Clearly, in some settings, various bargaining parties may not wish to reveal such information, but in some other settings, it can pave the way to a mutually agreeable negotiated agreement, just as market prices help to facilitate exchanges. Fifth, multicriteria approaches can help bypass at least some cognitive biases that are now well understood to adversely affect decisions. These models ask humans to do tasks for which they are best suited (specifying values) and then ask computers to do what they do best (numerical synthesis of information). These
A key limitation of multicriteria models is the lack of consensus on the best method to balance costs and benefits because benefit measures in these models purposefully include a wider array of attributes than do standard narrow measures such as qualityadjusted life-years (QALYs). A recent ISPOR Task Force report discussed three ways to approach these issues using multicriteria analysis [5]. The first uses direct inclusion of cost as a “negative” attribute with its own weight. The second involves finding other health care interventions (with known costs) that could be eliminated and assessing the multicriteria value of those as a benchmark. The third approach divides the multicriteria scores of each option by its cost (akin to using the ratio of cost per health benefit). The task force found none of these approaches perfectly persuasive, and urge further research on the matter. A fourth available alternative assumes an exogenous investment budget, without suggesting how to set its level [16]. We suggest a new alternative here. Cost-benefit analysis recommends adopting every alternative choice with positive net present value. This method, however, requires analysts to assign a specific value to human life, a task many resist. To solve this concern, cost-utility analysis calculates the incremental cost-effectiveness ratio (ICER), and the decision maker chooses the critical cutoff value for acceptable ICERs. We suggest a similar approach for use in multicriteria models. Suppose that in any multicriteria model, health-related measures (such as QALYs) accounted for a share S of the total value, and all other attributes accounted for a share of 1 S. Then, if the ICER cutoff was $100,000/QALY, the measured scores for each choice in the multicriteria model would be evaluated against a cutoff of $100,000/S. This would provide a direct method, comparable with that of cost-utility analysis when using an ICER cutoff to guide resource allocation in the broader framework of multicriteria decision analysis. Other concerns about multicriteria models are specific to particular applications. For example, the weight-setting protocol in the analytic hierarchy process allows internal inconsistencies and rank-reversal with different decision options [11]. The swing weight approach in the multi-attribute utility theory requires analysts to create standardized 0 to 100 scales for each attribute, which interact with the weights in potentially unforeseen ways [9]. These and other technical issues can and should be resolved to bring multicriteria models to their greatest potential.
Choices and Scenarios To illustrate key features of the multicriteria systems approach, consider a patient faced with choosing among available cancer therapies. Clinicians would specify the quantitatively measurable therapeutic and side-effect profiles of each option. Patients would specify the levels of any subjectively determined attribute associated with each choice, for example, distaste or fear of a
253
(0.148) (0.026) (0.012) (0.061) (0.012) 3 3 9 6 9 (0.026) (0.042) (0.314) (0.042) (0.148) 2 5 8 8 7
(0.203) (0.083) (0.026) (0.026) (0.042)
8 7 1 7 3
0 0 1 0 0 5,000 5,000 0 2,000 1,000
(0.314) (0.012) (0.111) (0.314) (0.061) 1 9 4 1 6 A B C D E
6 1 2 2 1
(0.061) (0.314) (0.203) (0.203) (0.314)
4 4 7 3 8
A. Treatment performance on nine attribute scales 0.8 0.3 2 50,000 0.2 0.8 8 50,000 0.3 0.4 5 80,000 0.8 0.7 3 20,000 0 0 10 2,000 B. Patient rankings of attributes (and associated weights) (0.111) 5 (0.083) 7 (0.042) 9 (0.012) (0.111) 3 (0.148) 2 (0.203) 6 (0.061) (0.042) 6 (0.061) 5 (0.083) 3 (0.148) (0.148) 4 (0.111) 5 (0.083) 9 (0.012) (0.026) 5 (0.083) 4 (0.111) 2 (0.203) 12 24 12 36 0
Probability of nausea (0–1.0) Probability of hair loss (0–1.0) Expected months of remission-free survival (0–36) Probability of remission (0– 0.5)
0.4 0.2 0.2 0.1 0
We also created five hypothetical patients with the same primary cancer who have different preference profiles and attribute ranks. The weights associated with these rankings come from the
T1 T2 T3 T4 T5
Patients
Treatment/ patient
Multi-attribute utility models require defining boundaries of the worst and best outcomes that can be imagined for the choices. Thus, for example, we set the best case for probability of remission at 0.5 (rather than 1.0), under the premise that for this disease, higher probabilities of remission cannot be imagined with existing treatment approaches. Table 1A reports the values of each of these five hypothetical therapies (T1–T5) on each of their nine attributes, showing (in the column headings) the feasible range and (where appropriate) which score is the best. In our example, all these attributes are quantified, whereas in other settings, some of the attributes might be subjectively determined, as previously discussed. T1 has the highest probability of remission, but lower overall survival if the regimen fails. It is supported by a moderate-sized randomized controlled trial (RCT), has mid-range costs to patient and society, and is quite likely to cause pain, hair loss, and considerable nausea. T2 is similar to T1, with a lower remission rate but potentially longer overall survival. It has stronger RCT evidence with higher expected months of remission-free survival. Costs are the same. It has less pain associated with it, but a higher chance of nausea. T3 is offered in an RCT. Preliminary evidence says it has high survival benefits and life extension, but is available to the patient only with 50% probability, and so the probabilities of remission and life extension reflect the expected value of the treatment and the control therapy in the RCT, and the side-effect profiles are a similar mix of the experimental and control therapies. T3 has a higher societal cost, but is free to the patient as part of the RCT. T4 has a low remission rate, but the highest expected remission-free survival. It is opposite of T1. It gives the most assurance for some additional survival, but little promise of remission. It has relatively low cost, and has strong RCT evidence supporting its outcomes. It has a side-effect profile that is unfavorable relative to other options. T5 is palliative care only. It offers no remission or life expectancy gains, but has no side effects and removes all pain associated either with therapy or with the disease itself. It costs little, and the data supporting its outcomes are solid.
Table 1 – Treatment performance and patient preferences based on multiple attributes.
Treatments
Pain scale (0–10)
Total cost ($1K–$100K)
Patient cost ($0–$10K)
Advancing knowledge (0 or 1)
Quality of evidence about benefits (0–5)
particular therapeutic modality (e.g., radiation, chemotherapy, or molecular targeting) independent of quantitatively known benefits and risks. As a specific example, we show the choice of five hypothetical cancer therapies for a set of hypothetical individuals using multiattribute utility theory, a specific approach within multicriteria systems analysis. This approach requires the decision maker (patient) to specify weights for the selected attributes. A simple method to obtain good weights simply asks the patient to list the desired attributes in rank order. Then, a formula—the rank order centroid method —estimates the average of all weights (summing to 1.0) consistent with the selected rankings [9]. This approximation has been shown to provide highly reliable results when used in linear multi-attribute models [10]. Our example uses rank order lists from the hypothetical patients to rank the importance of nine attributes in cancer therapy, which then create corresponding weights. The multi-attribute model combines the attribute performance measures of each therapy with the patients’ weights to create value scores on a 0 to 100 scale.
4 5 2 5 4
VALUE IN HEALTH 20 (2017) 251–255
254
VALUE IN HEALTH 20 (2017) 251–255
previously discussed rank order centroid method [9,10]. Table 1B summarizes each of these hypothetical patient’s preferences, showing the rank order in which they place each of the nine attributes of the therapies and (in parentheses) the associated rank order centroid weights. In short, our five patients have the following value structures. Patient A places the greatest value on the chance of a remission, and then out-of-pocket cost, and wants certainty about the quality of the evidence, followed by concerns about hair loss and nausea, freedom from pain, advancement of knowledge, and last, total societal cost. Patient B places the greatest value on expected months of remission-free survival, but has little concern about the probability of remission. This patient has heightened concerns about treatment side effects, modest concern for out-ofpocket costs and total costs, and little concern about advancing knowledge or the quality of data supporting the doctors’ recommendations. Patient C has a different view, committed to advancing societal knowledge, concerned about the total cost of care, but wishing to avoid side effects if possible. This patient has moderate interest in the probability of remission, and high interest in extending life expectancy, wishing primarily to survive until a granddaughter’s planned wedding. Patient D is self-oriented, caring only about survival, remission, pain, and side effects, and little at all about social or private cost, advancing knowledge, or the quality of the data. Patient E cares considerably about extending life expectancy, but is guilt-ridden about using resources that do not lead to remission, and so has great concern about the total cost of treatment, and also wishes to contribute to human knowledge about curing this disease. Modestly concerned about side effects, this patient has full insurance coverage and hence minimal concerns about out-of-pocket costs. Table 2 highlights each hypothetical patient’s most preferred choice of therapy. In some cases (e.g., patient A), the choices are close in value, whereas in other cases (e.g., patient B) they differ considerably. These scenarios illustrate our key point: preferences matter. Each of these hypothetical patient profiles could plausibly represent a real person and their trade-off considerations. Although these profiles were constructed deliberately to make our point, clearly the creation of preference weights that lead to different choices is not difficult. Indeed, our experience with using multi-attribute models in the realm of global health for prioritizing new vaccine development has led us to expect that differences in preference structure commonly lead to different outcomes in ranking of alternatives [17–19]. The values in Table 2 call to attention another feature of multi-attribute models: The scores of the different patients are unrelated to each other. Each patient’s preference structure creates a different value scale, just as, for example, Fahrenheit and Celsius temperature scales differ. Furthermore, these scales are not relative values. A value of 60 is not twice as good as a
Table 2 – Treatment scores based on patient values. Patient A B C D E
T1
T2
T3
T4
T5
58.6 37.4 32.0 51.1 37.1
56.0 59.3 39.1 54.8 49.5
58.5 52.5 59.4 50.0 48.8
50.3 58.5 42.8 44.5 59.8
54.9 61.6 36.4 42.5 46.7
Note. Figures in boldface represent the highest ranked treatment for each patient.
value of 30; it is 30 points better, just as 801F is not twice as warm as 401F, but is 401 warmer. These are interval, but not ratio, scales.
Choosing by Groups Although our hypothetical example has a single decision maker, the same conceptual approach remains valid in group decisionmaking processes. Groups would use one of several available voting methods to rank attributes in the model and to select (when necessary) appropriate values for any subjectively defined attributes. Each of these requires a collective choice, using a voting or grading process of some form, guidance for which to use is found in the social choice literature [20]. When might such issues appear? In health care, the occasions may involve investment choices, personnel decisions and related provision of care issues, or perhaps insurance coverage decisions. Suppose a prepaid managed care plan or an accountable care organization wished to choose the best cancer therapy for a particular condition. It might use samples of real patients to help make a system-wide choice among therapeutic options. Many other resource planning decisions have the same basic structure. The recent ISPOR Task Force reports provide useful examples of these types of multicriteria applications in group settings [4,5]. On the basic research end of the spectrum, review panels of governments or charitable foundations evaluate various grant proposals geared toward improving cancer therapy. Often in the current processes, peer reviewers give a composite numerical score to proposals under their consideration, with scores averaged to achieve final ranking recommendations. As an alternative, they could evaluate each proposal on the basis of various research attributes—uniqueness, likelihood of scientific breakthrough, public impact—and then use multicriteria models to provide final scoring of research proposals.
Conclusions Once various technical issues are resolved, we believe that multicriteria systems analysis models have much to favor them over alternative approaches used to guide health care choices, and possibly in broader program evaluation and comparisons. For decisions ranging from individual to societal, multicriteria models provide a valuable extension over single-criterion models such as cost-utility analysis. The participants in the health care enterprise (i.e., the 5Ps) commonly hold different perspectives, and so no single perspective can possibly meet all their needs. Using systems-based approaches improves transparency about their decision processes, because all assumptions and values are made specific. Often, these models can guide improvements in product design by varying attributes and costs of alternatives and measuring the value created by such variations. At least in some settings, this transparency can facilitate decision convergence among otherwise competing viewpoints. These models also help focus efforts to improve data in an efficient manner using sensitivity analysis. Finally, using these approaches also reduces risks of cognitive error associated with intuitive judgments and choices. In our view, these gains in assisting decision making at various levels provide compelling reasons to shift from narrow analytic techniques to systems approaches capable of incorporating a wide array of factors that actually drive real decisions in health and medicine.
VALUE IN HEALTH 20 (2017) 251–255
R EF E R EN CE S
[1] Garber A, Phelps C. Economic foundations of cost-effectiveness analysis. J Health Econ 1997;16:1–31. [2] Neumann PJ, Sanders GD, Russell LD, et al., editors. Cost-Effectiveness in Health and Medicine (2nd ed.). New York, NY: Oxford University Press, 2017. [3] Phelps C, Mushlin A. On the (near) equivalence of cost-effectiveness and cost-benefit analyses. Int J Technol Assess Health Care 1991;7:12–21. [4] Thokala P, Devlin N, Marsh K, et al. Multiple criteria decision analysis for health care decision making—an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health 2016;19:1–13. [5] Marsh K, IJzerman M, Thokala P, et al. Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health 2016;19:125–37. [6] Dolan J. Multi-criteria clinical decision support: a primer on the use of multiple criteria decision making methods to promote evidence based, patient-centered healthcare. Patient 2010;3:229–48. [7] Keeney RL, Raiffa H. Decisions with Multiple Objectives. Cambridge, UK: Cambridge University Press, 1993. [8] Keeney RL. Value Focused Thinking: A Path to Creative Decision Making. Cambridge, MA: Harvard University Press, 1992. [9] Edwards W, Barron FH. SMARTS and SMARTER—improved simple methods for multiattribute utility measurement. Organ Behav Hum Decis Process 1994;60:306–25.
255
[10] Barron FH, Barrett BE. Decision quality using ranked attribute weights. Manag Sci 1996;42:1515–23. [11] Saaty TL. Decision Making for Leaders: The Analytic Hierarchy Process for Decisions in a Complex World. Pittsburgh, PA: RWS Publications, 2008. [12] Phelps CE, Madhavan G, Rappuoli R, et al. Beyond cost-effectiveness: using systems analysis for infectious disease preparedness. Vaccine. In press. [13] Madhavan G, Phelps CE, Colwell R, et al. In plain view: a transparent systems approach for enhancing health policy decisions. Bridge (National Academy of Engineering) 2016;42:5–10. [14] Phelps CE, Mushlin AI. Focusing medical technology assessment using medical decision theory. Med Decis Making 1988;8:279–89. [15] Kahneman D. Thinking, Fast and Slow. New York, NY: Farrar, Straus and Giroux, 2011. [16] Phillips LD, Bana e Costa CA. Transparent prioritization, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing. Working Paper LSEOR 05.75, London School of Economics, 2005. [17] Phelps C, Madhavan G, Sangha K, et al. A priority-setting aid for new vaccine candidates. Proc Natl Acad Sci 2014;111:3199–200. [18] Madhavan G, Sangha K, Phelps C, et al., editors. Ranking Vaccines: A Prioritization Software Tool. Washington, DC: The National Academies Press, 2013. [19] Madhavan G, Phelps C, Rappuoli R, et al., editors. Ranking Vaccines: Applications of a Prioritization Software Tool. Washington, DC: The National Academies Press, 2015. [20] Pacuit E. Voting methods. 2012. Available from: http://plato.stanford.edu/ archives/win2012/entries/voting-methods/. [Accessed December 17, 2016].