- Email: [email protected]
A Comparison of National Guidelines for Network Meta-Analysis
-
Contents lists available at sciencedirect.com Journal homepage: www.elsevier.com/locate/jval
Policy Perspective
A Comparison of National Guidelines for Network Meta-Analysis Andrew Laws, MSc,1,* Ran Tao, BSc,2 Sisi Wang, MPH,2 Amie Padhiar, MSc,1 Sarah Goring, MSc3 1 Amaris Health Economics & Market Access, London, England, UK; 2ICON Plc, Vancouver, British Columbia, Canada; 3SMG Outcomes Research, Vancouver, British Columbia, Canada.
A B S T R A C T Background: Network meta-analysis (NMA) techniques allow the comparison of a complete set of interventions for patient groups. These meta-analysis techniques may be the only source of evidence to underpin estimates of comparative efficacy. Reimbursement agencies around the world are increasingly reliant upon meta-analysis techniques and assess their design and conduct with growing sophistication. Objective: Our objective is to create a superset of requirements collated from available national guidelines for the conduct of network meta-analysis such that a single analysis may be conducted to satisfy all reimbursement bodies that have specified requirements in sufficient detail. Methods: Published and draft guidelines documents from reimbursement bodies and health technology appraisal agencies were examined for their stipulations as to the conduct and design of network meta-analysis and measures to be taken to reduce bias and increase validity. Results: Guidelines from 41 countries were reviewed, yielding a sample size of 13 countries’ guideline documents: Australia, Belgium, Canada, China, Ireland, England and Wales, France, Germany, Scotland, South Africa, South Korea, Spain, Thailand, and a guideline document from the European Network of Health Technology Assessment, which contained explicit recommendations or requirements for conduct of NMA. Conclusion: This study expands the range of previous work that reviewed the guidelines for the use of indirect evidence from multiple national jurisdictions. These aggregate guidelines do not include requirements that are mutually prohibitive. It is possible to perform a single NMA for submission to an expanded list of national jurisdictions. Keywords: guidelines, meta-analysis, policy, reimbursement. VALUE HEALTH. 2019; 22(10):1178–1186
Introduction First introduced over a decade ago, network meta-analysis (NMA) is now a well-established approach for generating quantitative estimates of relative effect between multiple therapies.1–5 Around the world, results from NMAs inform decisions made by health technology assessment (HTA) agencies and recommendations in clinical practice guidelines. Despite the establishment of NMA as standard method for synthesizing evidence, there remains considerable variability in the quality of execution of NMA around the world. Recent systematic reviews of published NMAs found that quality, as assessed by both the Assessing the Methodological Quality of Systematic Reviews quality assessment tool and the quantitative tallying of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) domains, had not increased since 2010 despite the frequent publication of best-practice guidelines.6–10
A previous review comparing HTA submission guidelines for NMA, published in 2014, described NMA-related guidelines or methodological reviews issued by 9 national and international HTA agencies and organizations in Australia, Canada, South Africa, and 6 European countries.11 Because NMA methodology is continually being developed and refined, so too are HTA recommendations; several of the HTA bodies included in the previous review have since issued updated guidance documents. Furthermore, national HTA bodies that previously provided little or no guidance on NMAs have issued guidelines or methodological reviews that explicitly describe recommendations on the conduct and reporting of NMAs. The objectives of this review are 2-fold: first, to extract and describe an updated aggregate super-set of requirements from all available guideline documents, and second, to determine if any of those requirements are mutually exclusive and therefore preclude the possibility of performing a single NMA for submission to multiple jurisdictions.
* Address correspondence to: Andrew Laws, BSc, MSc, PgDip, Amaris HEMA, 13 Hawley Crescent, London, England, UK NW1 8NP. Email: andrewlaws.contact@gmail. com 1098-3015/$36.00 - see front matter Copyright ª 2019, ISPOR–The Professional Society for Health Economics and Outcomes Research. Published by Elsevier Inc. https://doi.org/10.1016/j.jval.2019.05.013
POLICY PERSPECTIVE
1179
Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses diagram.
Methods Identification of Relevant Documents This review used the same sampling frame as the original study; the website of country-specific pharmacoeconomic guidelines maintained by ISPOR at https://tools.ispor.org/peguidelines/.12 As of the November 27, 2017, this comprised guidelines from 41 countries or regions, of which 33 were considered in the original study: Australia, Austria, Baltic States (Latvia, Lithuania, Estonia), Belgium, Brazil, Canada, Colombia, China, Croatia, Cuba, Czech Republic, Denmark, Egypt, England and Wales, the European Union, Finland, France, Germany, Hungary, Ireland, Israel, Italy, Japan, Malaysia, Mexico, New Zealand, The Netherlands, Norway, Poland, Portugal, Russian Federation, Scotland, Slovenia, South Africa, South Korea, Spain, Spain (Catalonia region), Sweden, Thailand, Taiwan, and the United States.13–54
For all documents included in the original study, and all documents identified in the ISPOR repository, the corresponding national health authority website was searched to ensure the document was the most recent available version, or to update with a more recently issued version. Each document was reviewed for its eligibility for this study. Documents were required to be either an HTA-related guideline or methodological review issued by an HTA body, to have provided sufficient documentation pertaining to NMA conduct to enable comparison, and to be available in English, Cantonese, or Mandarin.
Comparison of National Guidelines Extraction checklists were developed to summarize recommendations in the identified national guidelines, spanning the full breadth of NMA conduct, from identifying the evidence base to the methodological and statistical aspects of the quantitative evidence synthesis. The checklists were separated into the following
1180
VALUE IN HEALTH
OCTOBER 2019
Table 1. Recommendations regarding the conduct and reporting of literature review.
Design, conduct, and reporting of trial searches
Australia
Report date(s) search conducted
Specify search date span
Specify PICO criteria
Belgium
Justify exclusion criteria
Describe search terms and relationship
Present strategy in blocks by indication, intervention, study type, etc.
Perform supplementary searches
Present PRISMA diagram
Canada
Follow CRD systematic review procedures
China Ireland England & Wales
EUnetHTA France Germany
Scotland
South Africa
South Korea
Spain
Thailand
CENTRAL indicates Cochrane Central Register of Controlled Trials; CINAHL, Cumulative Index to Nursing and Allied Health Literature; CRD, Centre for Reviews and Dissemination; EED, Economic Evaluation Database; EUnetHTA, European Network of Health Technology Assessment; HEED, Health Economic Evaluation Database; ICTRP, International Clinical Trials Registry Platform; NHS, National Health Service; PICO, Patient-Intervention-Comparator-Outcome; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses. * eg, Australia New Zealand Clinical Trials Registry (formerly the Australia Clinical Trials Registry); Arzeneitmittel-information system; Pharmnet.BUND Klinische Prüfung. † eg, PsycINFO (psychology and psychiatry), MANTIS (osteopathy and chiropractic).
conceptual subgroups: (1) conduct and reporting of literature search, (2) study selection, (3) reporting and assessment of included studies, and (4) NMA. The review checklists were created through an iterative process designed to capture all recommendations and requirements. Where a national guideline made a recommendation, the associated item was added to the checklist unless already present. Uncertainty as to the interpretation of a recommendation, and whether concept duplication had occurred, was resolved through discussion with a second and, if required, third reviewer. Although time-consuming and broad in scope, this allowed us to capture every recommendation without judgment as to their methodological worth. Each checklist was completed for each of the guidelines by 2 separate reviewers. A final review of the guidelines was conducted, and any additional items required were added to the checklists where applicable. Finally, the checklists were compared across reviewers and any discrepancies were resolved through discussion with a third reviewer. Throughout the assessments, we did not differentiate between a recommendation and a requirement because the language used within the assessed documentation made this difficult to discern.
Results Of the 41 documents assessed, only guidelines documents from Australia, Belgium, Canada, China, England and Wales, France, Germany, Ireland, Scotland, South Africa, South Korea, Spain, and
Thailand made recommendations as to the conduct of NMAs.13–25 These guidelines, along with a guidance document issued by the European Network for Health Technology Assessment,26 comprise recommendations from 14 jurisdictions across Europe, Asia, North and South America, and Africa. Of the excluded guidelines, 10 documents mentioned the use of NMA but did not provide any recommendations as to the conduct of the analysis; these were issued by agencies in Croatia, Egypt, Hungary, Japan, The Netherlands, New Zealand, Norway, Poland, Sweden, and the United States. Nine documents were excluded because they made no mention of NMA techniques; these were issued by agencies in Austria, Denmark, Baltic States (Estonia, Latvia, and Lithuania), Finland, Israel, Italy, Colombia, Malaysia, and Taiwan. A further 7 documents were excluded because they were presented in a language other than English, Mandarin, or Cantonese (Fig 1, Appendix Table 1, see Supplemental Materials found at https://doi. org/10.1016/j.jval.2019.05.013).
Conduct and Reporting of Literature Search Table 1 summarizes the recommendations provided in the guidelines and methods documents relating to how the search for evidence was executed and reported. Of the 14 HTA bodies whose guideline documents were included in the review, 12 address some aspect of the literature search; neither China16 nor Ireland19 provide such guidance. Regarding the design, conduct, and reporting of the searches, there is an overall focus on repeatability; presenting the search
POLICY PERSPECTIVE
1181
Table 1. Continued Specified databases Medline
Embase
Cochrane (CENTRAL)
Cochrane (Systematic reviews)
Clinicaltrials. gov
Clinical trial register of EMA
Jurisdictionspecific databases*
CINAHL
ICTRP search portal
NHS CRD
Subject-specific databases or registers†
Manufacturers internal databases
Table 2. Study selection. Selection process
Trials include Maintained log of trials deemed ineligible
Trials form a single connected network of interventions
Requirement for homogeneity of prognostic severity
Predefined inclusion/ exclusion criteria process
Justified selection criteria
Australia
Belgium
Selected by 2 independent reviewers
England & Wales
France
Germany
Scotland
South Africa
Homogeneity between trials
EUnetHTA
Blinded outcome assessment
China Ireland
Blinded trial (participants/ personnel)
Canada
Randomized allocation*
South Korea Spain
Thailand EUnetHTA indicates European Network of Health Technology Assessment. *Criteria for randomized controlled trials.
1182
VALUE IN HEALTH
OCTOBER 2019
Table 3. Reporting and assessment of included studies. Reporting and assessment of clinical trials Describe the design and methodology according to CONSORT guidelines
Assessment of variance between trial protocol and standard practice
Assess adequacy of blinding
Comparison Implementation Assessment Describe Investigator of of ITT of time conflicts of rates of difference horizon interest dropout/ in baseline reported loss to risk and follow-up placebo response
Report of subgroup analysis
Treatment effect modifiers identified prior to comparing study results
Assessment of publication bias and/or funnel plot
Australia
Belgium
Canada
China Ireland
England & Wales
EUnetHTA
France Germany
Scotland
South Africa
South Korea
Spain
Thailand
CONSORT indicates Consolidated Standards of Reporting Trials; EUnetHTA indicates European Network of Health Technology Assessment; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; ITT, intent-to-treat.
Table 4. Conduct of network meta-analysis.
Australia
Description of patient & treatment characteristics
Describe & justify statistical method; Bayesian or Frequentist.
If Bayesian, describe prior distribution, sensitivity to priors, and assess convergence.
Present Individual study results and/or Forest plot
Identify and report influential studies using metrics (residuals, Cooks distance)
Graphically summarize rank probabilities
Description of relative-effect estimate
Include rationale for, and description of, sensitivity analyses
Description of different findings with sensitivity / scenario analysis
Belgium Canada
China
Ireland England & Wales
EUnetHTA
France
Germany
Scotland
South Africa
South Korea
Thailand
pain
POLICY PERSPECTIVE
strategies systematically; the use of specified patient, intervention, comparator, outcome criteria; and conducting supplemental searches including manual checking of the references in retrieved papers and gray literature searches. Only the Belgian guidelines specify that the search should comply with best-practice guidelines from a specific organization (specifically, the Centre for Reviews and Dissemination).55 Across the documentation, Australia, Belgium, Germany, and Scotland provide the largest number of specific requirements for the conduct of trial searches, and all 4 specify the following 2 requirements: (1) that search terms, such as medical subject heading identifiers, and other keywords used to search electronic databases should be described; and (2) that such descriptors are grouped into related blocks in the presentation of the search strategy.13,14,19,21 For the bibliographic databases and registries to be searched (Table 1), only half of the HTA bodies provide specific recommendations. Of those that do provide guidance, searches of Medline and EMBASE are required by all of the 7, and search of the Cochrane Central Registry of Controlled Trials is specified by 5 of these 7. Many guideline documents require additional searching in jurisdiction-specific clinical trial databases. The German guidelines require that searches include the German-language Arzeneitmittel-informations system and PharmnetBUND Klinische Prüfung, and the national guideline document issued by the Australian Pharmaceutical Benefits Advisory Committee requires that the Australia New Zealand Clinical Trials Registry form part of the search strategy.13,19 Guidelines issued by Australia, England and Wales, Germany, and Scotland require that company-specific databases be searched and results presented in the submission, although the guideline documents contain no indication of how the validity of such a search would be confirmed.13,17,19,21
1183
considerable biasing of results may occur through the selection of trials with varying levels of quality. Most of the guideline documents specify that inclusion or exclusion criteria for trials in the NMA should be defined before the selection process, and that selection criteria be justified in the submission (Table 2). In addition, 6 of the 14 guideline documents require that a record of trials that are deemed ineligible is presented alongside the trials that meet selection criteria, an important step in reducing the risk of selection bias and increasing the level of transparency. Nine of the 14 guideline documents found it necessary to explicitly state that random allocation to study arms was a requirement for inclusion in NMAs, and 6 documents specified that trials should be double-blinded (Table 2).
Reporting and Assessment of Included Studies Elements relating to the transparent reporting and critical appraisal of studies contributing to NMAs are summarized in Table 3, capturing the recommendations relating to this aspect of evidence synthesis provided by 12 of the 14 HTA bodies included in the review. There was consistency across the guideline documents, with most requiring an assessment of the level of bias within studies, an assessment of between-study differences in baseline risk and placebo response, and a description of the time horizon for the studies. Only Ireland and Germany specified that design and methodology were described according to Consolidated Standard of Reporting Trials guidelines.56 The requirement to provide a report of subgroup analysis was the most consistently applied item, and 5 guidelines required the use of a funnel plot to assess bias.
Conduct of NMA Study Selection The selection of the studies that will form the network of trials is a key part of the conduct of an NMA, and a step where
Table 4 describes the required statistical techniques, processes, and presentation of the NMA. No single aspect of the conduct of NMA is universally addressed by all HTA bodies. The most
Table 4. Continued MetaNaïve regression comparisons of point estimates or active arms are prohibited
Include analysis of the hypothesis of consistency and/or transitivity assumption
Assess the results for each common reference across trials for any important differences
Perform consistency check between direct and indirect evidence
Assess homogeneity of direct comparisons
Assess heterogeneity with Cochrane Q, I2
Compare Random Include the results effects diagram from the model of simple indirect network comparison structure method (Buchers difference)
Assess level of bias within included studies
Provide annotated software code
1184
VALUE IN HEALTH
frequently required features of NMA included the presentation of a random effects model, the justification of using either frequentist or Bayesian statistics, the assessment of homogeneity of direct comparisons, and the use of sensitivity analysis. None of the guideline documents stipulated what form sensitivity analyses should take, but 4 of the HTA bodies require a rationale for sensitivity analyses, 6 countries subsequently required a description of the different findings, and 4 needed a relative effect estimate. Most HTA bodies call for a consistency check between direct and indirect evidence, with the Australian guidelines suggesting the use of Bucher’s method, indicating that indirect evidence should be considered alongside direct evidence and the National Institute for Health and Care Excellence Decision Support Unit documentation providing details on the use of inconsistency models. The National Institute for Health and Care Excellence Decision Support Unit documentation provides suggestions on how to deal with inconsistency, if detected, suggesting re-evaluation of the choice of scale, effect modification, and similarity of the contributing randomized controlled trials. The documentation provides questions for critical evaluation of the source and impact of inconsistency but acknowledges that it may also be due to chance. None of the other national guidelines indicated that differences between direct and indirect evidence would render indirect evidence inadmissible. England and Wales was the only country to require the inclusion of code written for analysis and the statistical software packages used.17 Almost half the countries explicitly prohibited the use of naïve indirect comparisons, where the results of individual arms of different trials are compared as though they were from the same trial. Appendix Table 2 (see Supplemental Materials found at https://doi.org/10.1016/j.jval.2019.05.013) shows the various ways of presenting the results of an NMA and choices of scale for analysis. The various national guideline documents appear to take 2 approaches to the issue of data scales in the presentation of results: either to present a wide-ranging set of scales or to not present any requirements at all and leave the choice of scale the judgment of those conducting the NMA. Ireland requires that both absolute and relative effect scales are presented, but the specific choice of scale (eg, odds ratio, relative risk, or hazard ratio) is not addressed.
Discussion This study builds on a previous comparison of multiple national guideline documents.11 Using a similar study framework, the limited scope of the original study has been expanded to include additional countries and to include guidelines that have been updated. Compliance with the PRISMA statement is highly variable among the component guidelines of the national guidelines superset, with guidance on systematic review being outside the scope of countries such as Canada, China, and South Korea. Nevertheless, when considered as a compiled super-set of requirements, the national guidelines compare favorably to the PRISMA extension statement on the reporting of NMA. Of the domains on the PRISMA checklist related to conduct of study search, selection, and conduct of NMA, only items related to description of data extraction forms and reporting of methods used to explore the geometry of the network are lacking from the national guidelines super-set. The lack of requirements to assess biases implicit in network geometries is a notable omission because the PRISMA statement includes 3 items on its 27-domain checklist about the reporting of network structures and testing of alternate geometries.
OCTOBER 2019
The criteria for conducting NMA can broadly be categorized into the following: assessments and analyses to test assumptions required for a NMA, presentation and reporting of results, and justification for modeling choices. Where companies have limited time to conduct analyses, the criteria expected to have the largest impact on the results and conclusions for the NMA would be those assessing the assumptions of a NMA and, more specifically, the assessment of heterogeneity between studies. An example of a criterion that may not be impactful is requirement to test the sensitivity of the model results to the choice of prior for a Bayesian NMA. It may be time-consuming to perform this sensitivity check, and typically for models that converge well with sufficient number of studies and data points, it could be argued that this may not be necessary. Some of the criteria would not be expected to affect the conduct of an NMA and therefore may not be as necessary to meet. For instance, it may not be necessary to graphically summarize the rank probabilities, which are only required for England and Wales, and Germany. Nevertheless, considering that the ranking probabilities can easily be obtained from software such as WinBUGs (commonly used for Bayesian NMAs), exclusion of this step is unlikely to generate efficiencies. In addition, testing for consistency is typically conducted as a last step of the NMA process. Many published NMAs do not report the output of such assessment, although this is required for 8 of the 14 countries. Inconsistency in NMA results are usually explainable based on heterogeneity identified between studies. Therefore, if heterogeneity is appropriately assessed and adjusted for where feasible, this should lead to fewer inconsistencies being observed or at least the ability to explain some of the inconsistencies identified. Our assessment is currently the only attempt to collect together the disparate requirements for the conduct of NMA that the various national jurisdictions expect of reimbursement submissions. For the first time, this included countries from outside the Europe and commonwealth nations. China, South Korea, and Thailand all indicated their respective requirements for conduct of NMA in sufficient detail to allow them to be included.16,22,23,25 The limitations of this study arise from 3 sources: access to the source documents, limitations in the source documents themselves, and weaknesses in the review process. A limitation in accessing national guidelines is the lack of a searchable database. This review searched documents from a variety of sources, all of which were based online; however, the lack of a definitive repository for national guidelines means that the authors cannot be certain that all eligible guideline documents were captured by our search. Furthermore, online repositories that appeared comprehensive often hosted out-of-date versions of guideline documents or linked to pages that were no longer available, highlighting the difficulties involved in maintaining an online collection of living documents in the digital era. A further limitation is the effect of language restriction on the inclusion of guideline documents. Guidelines from Brazil, Iran, Mexico, and Cuba were identified but could not be appraised owing to lack of English language translations. The limitations of the guideline documents themselves were a lack of a shared vocabulary of nontechnical terms. Whilst techniques such as funnel plot or use of Cook’s distance are unambiguous and language-independent, nontechnical requirements may be interpreted subjectively and placed in the subcategory of the checklist where they best fit. The compounding effects of our subjective interpretations may be that readings of the guideline documents became informed by the guideline checklist tables themselves; however, our iterative, consensus-building approach was designed to minimize such bias.
POLICY PERSPECTIVE
Further work should focus on the requirements from emerging markets in South America, Africa, and South Asia and documentation relating to emerging techniques in evidence synthesis such as matching adjusted indirect comparisons, simulated treatment comparisons, and methods being used in the synthesis of nonrandomized evidence. This study expands the range of previous work that reviewed the guidelines for the use of indirect evidence from multiple national jurisdictions. These updated aggregate guidelines do not include requirements that are mutually prohibitive. It is possible to perform a single NMA for submission to a variety of national jurisdictions.
17.
Acknowledgments
21.
The authors have no other financial relationships to disclose.
18.
19.
20.
22.
Supplemental Material 23.
Supplementary data associated with this article can be found in the online version at https://doi.org/10.1016/j.jval.2019.05.013.
REFERENCES 1. 2. 3.
4.
5. 6.
7.
8.
9.
10.
11. 12.
13.
14.
15.
16.
Ioannidis J. Indirect comparisons: the MESH and MESS of clinical trials. Lancet. 2006;368:1470–1472. Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ. 2005;331:897–900. Bucher HC, Guyatt GH, Griffith LE, et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997;50:683–691. Song F, Altman DG, Glenny AM, et al. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ. 2003;326:472. Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med. 2004;23:3105–3124. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:2012. Hutton B, Salanti G, Caldwell DM, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med. 2015;162(11):777–784. Hoaglin DC, Hawkins N, Jansen JP, et al. Conducting indirect-treatmentcomparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health. 2011;14:429–437. Jansen JP, Fleurence R, Devine B, et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health. 2011;14:417–428. Jansen JP, Trikalinos T, Cappelleri JC, et al. Indirect treatment comparison/ network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17:157–173. Laws A, Kendall R, Hawkins N. A comparison of national guidelines for network meta-analysis. Value Health. 2014;17:642–654. International Society for Pharmacoeconomics and Outcomes Research. Pharmacoeconomics guidelines around the world. 2012. https://tools.ispor. org/peguidelines/. Accessed November 27, 2017. Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee, version 5.0. 2016. https://pbac.pbs.gov.au/. Accessed November 27, 2017. Cleemput I, Neyt M, van de Sande S, Thiry N. Belgian guidelines for economic evaluations and budget impact analyses: second edition. Health Technology Assessment (HTA). Belgian Health Care Knowledge Centre. https://kce.fgov. be/sites/default/files/atoms/files/KCE_183_economic_evaluations_second_ed ition_Report_update.pdf. Accessed November 27, 2017. Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada – 4th edition. https://www.cadth.ca/dv/guidelines-economic-evaluation-health-technologi es-canada-4th-edition. Accessed November 27, 2017. China Center for Health Economics Research. China guidelines for pharmacoeconomic evaluation. 2011. https://tools.ispor.org/PEguidelines/ source/China-Guidelines-for-Pharmacoeconomic-Evaluations_2011_Chinese. pdf. Accessed November 27, 2017.
24. 25.
26.
27.
28. 29.
30.
31.
32. 33.
34.
35.
36.
37.
38.
39.
1185
National Institute for Health and Care Excellence. Guide to the methods of technology appraisal 2013. https://www.nice.org.uk/process/pmg9/resourc es/guide-to-the-methods-of-technology-appraisal-2013-pdf-2007975843781. Accessed November 27, 2017. Haute Autorite de Sante. Choices in Methods for Economic Evaluation. Department of Economic and Public Health, ed. Saint-Denis La Plaine, France; 2012. https://www.has-sante.fr/upload/docs/application/pdf/2012-1 0/choices_in_methods_for_economic_evaluation.pdf. Accessed November 27, 2017. Institute for Quality and Efficiency in Health Care. General methods (version 4.2) of 22 April 2015. Health Information and Quality Authority. Guidelines for the economic evaluation of health technologies in Ireland 2014. https://www.hiqa.ie/sites/default/files/2017-01/Revised_Eco nomic_Guidelines_posted_100714.pdf. Accessed November 27, 2017. South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. http://www.gpw online.co.za/Gazettes/Gazettes/36118_1-2_Health.pdf. Accessed November 27, 2017. Scottish Medicines Consortium. Guidance to Manufacturers for Completion of New Product Assessment Form (NPAF) 2017.22. https://www.scottish medicines.org.uk/making-a-submission/. Accessed November 27, 2017. National Evidence-based Healthcare Collaborating Agency (NECA). The Application of Bayesian Meta-Analysis in HTA: The Experience from Korea. 2014. https://www.neca.re.kr/report_download.do?seq=38&key=1&cms_menu_seq= 141. Accessed November 27, 2017. Bae S, Lee S, Bae EY, et al. Korean guidelines for pharmacoeconomic evaluation (second and updated version): consensus and compromise. Pharmacoeconomics. 2013;31:257–267. Ortega AF, Fraga MD, Alegre-del-Ray EJ, et al. A checklist for critical appraisal of indirect comparisons. Int J Clin Pract. 2014;68(10):1181–1189. Chaiyakunapruk N, Saokaew S, Sruamsiri R, et al. Systematic review and network meta-analysis in health technology assessment. J Med Assoc Thai. 2014;97(suppl 5):S33–42. European Network of Health Technology Assessment (EUnetHTA). EUnetHTAJA1 WP5 - comparators & comparisons: direct and indirect comparisons. 2015. https://www.eunethta.eu/wp-content/uploads/2018/01/ Comparators-Comparisons-Direct-and-indirect-comparisons_AmendedJA1-Guideline_Final-Nov-2015.pdf. Accessed November 27, 2017. Agency for Quality and Accreditation in Health Care Croatia. The Croatian Guideline for Health Technology Assessment Process and Reporting. 2011. http:// aaz.hr/sites/default/files/hrvatske_smjernice_za_procjenu_zdravstvenih_ tehnologija.pdf. Accessed November 27, 2017. Elsisi GH, Kalo Z, Eldessouki R, et al. Recommendations for reporting pharmacoeconomic evaluations in Egypt. Value Health Reg Issues. 2013;2(2):319–327. Ministry of Human Capacities - State Secretariat for Health. Professional healthcare guideline on the methodology of health technology assessment. Gyogyszereink. 2017;67(1). Central Social Insurance Medical Council. Guideline for preparing cost-effectiveness evaluation to the Central Social Insurance Medical Council. 2016. https://tools.ispor.org/PEguidelines/source/Japanese_PE_Guideline.pdf. Accessed November 27, 2017. Ijzerman MJ, Al MJ, de Boer A, et al. Guideline for economic evaluations in healthcare. Zorginstituut Nederland. 2016. https://tools.ispor.org/PEguide lines/source/Netherlands_Guideline_for_economic_evaluations_in_healthcare .pdf. Accessed November 27, 2017. Pharmaceuticals Management Agency (PHARMAC). Guidelines for funding applications to PHARMAC; 2017. Legemiddelverk Statens. Guidelines on how to conduct pharmaco economic analyses. 2012. https://tools.ispor.org/PEguidelines/source/ Norwegian_guidelines2012.pdf. Accessed November 27, 2017. Agencja Oceny Technologii Medyczncyh. Guidelines for Conducting Health Technology Assessment (HTA). AOTM; 2009. https://tools.ispor.org/ PEguidelines/countrydet.asp?c=16&t=2. Accessed November 27, 2017. Tandvårds-och läkemedelsförmånsverket. Handbok till Tandvårds- och läkemedelsförmånsverkets föreskrifter (TLVFS 2011:3) om ansökan om pris och subvention för förbrukningsartiklar Sweden. 2011. https://www.tlv.se/ download/18.467926b615d084471ac3390c/1510316399612/Handbok-TLVFS2 011-3-subvention-forbrukningsartiklar-sept-2011.pdf. Accessed November 27, 2017. Academy of Managed Care Pharmacy (AMCP). The ACMP Format for Formulary Submissions Version 4.0. Alexandria, VA: Academy of Managed Care Pharmacy; 2016. https://www.amcp.org/sites/default/files/2019-03/AMCPFormat-V4.pdf. Accessed November 27, 2017. Walter E, Zehetmayr S. Guidelines on Health Economic Evaluation. Vienna, Austria: Institut fur Pharmaokonomische Forschung; 2006. http://www.ipfac.at/fileadmin/template/PDF%20alt/Konsens_Guidelines_en.pdf. Accessed November 27, 2017. Kristensen FS H, Andersen SE, Andersen S, Bonnevie B. Health Technology Assessment Handbook. Denmark: Danish Centre for Health Technology Assessment, National Board of Health; 2007. https://www.sst.dk/w/media/ ECAAC5AA1D6943BEAC96907E03023E22.ashx. Accessed November 27, 2017. Behmane D, Lambot K, Irs A, et al. Baltic Guideline for Economic Evaluation of Pharmaceuticals. Medicines Pricing & Reimbursement Agency, Latvia; Ministry of Health, Lithuania; Health Insurance Fund, Estonia; Ministry of Social
1186
40.
41.
42.
43.
44.
45.
46.
47.
VALUE IN HEALTH
Affairs, Estonia; 2002. https://tools.ispor.org/PEguidelines/source/Baltic-PEguideline.pdf. Accessed November 27, 2017. Lääkkeiden Hintalautakunta. Preparing a Health Economic Evaluation to Be Attached to the Application for Reimbursement Status and Wholesale Price for a Medicinal Product: Application Instructions. Finland: Lääkkeiden Hintalautakunta; 2017. http://www.hila.fi/c/document_library/get_file?folderId=1133 981&name=DLFE-9844.pdf. Accessed November 27, 2017. Ministry of Health (Israel) - Pharmaceutical Administration. Guidelines for the Submission of a Request to Include a Pharmaceutical Product in the National List of Health Services - Appendix D: Instructions for Performing an Economic Assessment. Israel: Ministry of Health; 2010. https://tools.ispor.org/PEguide lines/source/Israel-Guidelines-for-submission_2010.pdf. Accessed November 27, 2017. Capri S, Ceci A, Terranova L, et al. Guidelines for economic evaluations in Italy: recommendations from the Italian group of pharmacoeconomic studies. Drug Inf J. 2001;35(1):189–201. Faria R, Duarte A, McKenna C, Revill P, Cary M, Byford S. Guidelines for the economic evaluation of healthcare technologies in Colombia: technical support documents. Bogota, Colombia: Instituto de Evaluacion Tecnologica en Salud; 2014. http://www.iets.org.co/Archivos/67/Documentos-tecnicosespanol.pdf. Accessed November 27, 2017. Ministry of Health Malaysia, Pharmaceutical Services Division - MIH Technical Working Group. Pharmacoeconomic Guideline for Malaysia. 2012. https://www.pharmacy.gov.my/v2/en/documents/pharmacoeconomic-guide line-malaysia.html. Accessed November 27, 2017. Taiwan Society for Pharmacoeconomics and Outcomes Research (TaSPOR). Guidelines of Methodological Standards for Pharmacoeconomic Evaluations in Taiwan (version 1.0). Taiwan. 2006. https://tools.ispor.org/PEguidelines/ source/2006_PEG_EN_2009.pdf. Accessed November 27, 2017. Ministerio da Saude. Diretrizes Metodologicas. Diretriz de Avaliacao Economica. 2nd ed. Brazil: Ministerio da Saude; 2014. http://bvsms.saude.gov.br/bvs/ publicacoes/diretrizes_metodologicas_diretriz_avaliacao_economica.pdf. Accessed November 27, 2017. Ministry of Public Health. Methodological Guidelines for Health Economic Evaluation. Cuba: Ministry of Public Health; 2003. http://scielo.sld.cu/scielo.php? script=sci_arttext&pid=S0864-34662004000100005. Accessed November 27, 2017.
OCTOBER 2019
48.
49.
50.
51.
52.
53.
54.
55.
56.
State Institute for Drug Control (SUKL). Cost-effectiveness Guidelines. Czech Republic: State Institute for Drug Control (SUKL); 2017. http://www.sukl.cz/ file/85481_1_1. Accessed November 27, 2017. National Institute of Health Research of the Islamic Republic of Iran. Guidelines for the Formation of HTA Proposal. Iran: National Institute of Health Research of the Islamic Republic of Iran; 2017. http://nihr. tums.ac.ir/upfiles/documents/307113275.pdf. Accessed November 27, 2017. Consejo de Salubridad General. Economic Assessment Study Guideline for Updating the National Formulary in Mexico. Mexico: Consejo de Salubridad General; 2015. http://www.csg.gob.mx/descargas/pdf/priorizacion/cuadrobasico/guias/insumos_salud/GEI_2017_Diciembre.pdf. Accessed November 27, 2017. Center for Healthcare Quality Assessment and Control of the Ministry of Health of the Russian Federation. Guidelines for Conducting a Comparative Clinical and Economic Evaluation of Drugs. Russia: Center for Healthcare Quality Assessment and Control of the Ministry of Health of the Russian Federation; 2016. https://rosmedex.ru/wp-content/uploads/2016/12/MR-KE% 60I-23.12.2016.pdf. Accessed November 27, 2017. Mora-Ripoli R, Gilabert-Perramon A, Oliva-Moreno J, Puig-Junoy J. Guidance for economic evaluation and budget impact analysis for pharmaceuticals in Catalonia (Spain). Value Health. 2014;17(7):A447. Health Insurance Institute of Slovenia. Rules on Classification of Medicines on the List, Accepted by the Health Insurance Institute of Slovenia. 2013. https:// www.farmaforum.si/sites/43/files/files/Pravilniki/Pravilniki%20ENG/Rules%20of% 20reimbursment.pdf. Accessed November 27, 2017. Alves da Silva E, Gouveia Pinto C, Sampaio C, Pereira JA, Drummond M, Trindade R. Guidelines for Economic Drug Evaluation Studies. Lisbon, Portugal: Autoridade Nacional do Medicamento e Produtos de Saude; 1998. http://www.infarmed.pt/portal/page/portal/INFARMED/ENGLISH/PCAEC04_ vering.pdf. Accessed November 27, 2017. University of York. Centre for Reviews and Dissemination. Systematic Reviews. CRD’s Guidance for Undertaking Reviews in Health Care. York, England: University of York; 2009. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.
Contents lists available at sciencedirect.com Journal homepage: www.elsevier.com/locate/jval
Policy Perspective
A Comparison of National Guidelines for Network Meta-Analysis Andrew Laws, MSc,1,* Ran Tao, BSc,2 Sisi Wang, MPH,2 Amie Padhiar, MSc,1 Sarah Goring, MSc3 1 Amaris Health Economics & Market Access, London, England, UK; 2ICON Plc, Vancouver, British Columbia, Canada; 3SMG Outcomes Research, Vancouver, British Columbia, Canada.
A B S T R A C T Background: Network meta-analysis (NMA) techniques allow the comparison of a complete set of interventions for patient groups. These meta-analysis techniques may be the only source of evidence to underpin estimates of comparative efficacy. Reimbursement agencies around the world are increasingly reliant upon meta-analysis techniques and assess their design and conduct with growing sophistication. Objective: Our objective is to create a superset of requirements collated from available national guidelines for the conduct of network meta-analysis such that a single analysis may be conducted to satisfy all reimbursement bodies that have specified requirements in sufficient detail. Methods: Published and draft guidelines documents from reimbursement bodies and health technology appraisal agencies were examined for their stipulations as to the conduct and design of network meta-analysis and measures to be taken to reduce bias and increase validity. Results: Guidelines from 41 countries were reviewed, yielding a sample size of 13 countries’ guideline documents: Australia, Belgium, Canada, China, Ireland, England and Wales, France, Germany, Scotland, South Africa, South Korea, Spain, Thailand, and a guideline document from the European Network of Health Technology Assessment, which contained explicit recommendations or requirements for conduct of NMA. Conclusion: This study expands the range of previous work that reviewed the guidelines for the use of indirect evidence from multiple national jurisdictions. These aggregate guidelines do not include requirements that are mutually prohibitive. It is possible to perform a single NMA for submission to an expanded list of national jurisdictions. Keywords: guidelines, meta-analysis, policy, reimbursement. VALUE HEALTH. 2019; 22(10):1178–1186
Introduction First introduced over a decade ago, network meta-analysis (NMA) is now a well-established approach for generating quantitative estimates of relative effect between multiple therapies.1–5 Around the world, results from NMAs inform decisions made by health technology assessment (HTA) agencies and recommendations in clinical practice guidelines. Despite the establishment of NMA as standard method for synthesizing evidence, there remains considerable variability in the quality of execution of NMA around the world. Recent systematic reviews of published NMAs found that quality, as assessed by both the Assessing the Methodological Quality of Systematic Reviews quality assessment tool and the quantitative tallying of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) domains, had not increased since 2010 despite the frequent publication of best-practice guidelines.6–10
A previous review comparing HTA submission guidelines for NMA, published in 2014, described NMA-related guidelines or methodological reviews issued by 9 national and international HTA agencies and organizations in Australia, Canada, South Africa, and 6 European countries.11 Because NMA methodology is continually being developed and refined, so too are HTA recommendations; several of the HTA bodies included in the previous review have since issued updated guidance documents. Furthermore, national HTA bodies that previously provided little or no guidance on NMAs have issued guidelines or methodological reviews that explicitly describe recommendations on the conduct and reporting of NMAs. The objectives of this review are 2-fold: first, to extract and describe an updated aggregate super-set of requirements from all available guideline documents, and second, to determine if any of those requirements are mutually exclusive and therefore preclude the possibility of performing a single NMA for submission to multiple jurisdictions.
* Address correspondence to: Andrew Laws, BSc, MSc, PgDip, Amaris HEMA, 13 Hawley Crescent, London, England, UK NW1 8NP. Email: andrewlaws.contact@gmail. com 1098-3015/$36.00 - see front matter Copyright ª 2019, ISPOR–The Professional Society for Health Economics and Outcomes Research. Published by Elsevier Inc. https://doi.org/10.1016/j.jval.2019.05.013
POLICY PERSPECTIVE
1179
Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses diagram.
Methods Identification of Relevant Documents This review used the same sampling frame as the original study; the website of country-specific pharmacoeconomic guidelines maintained by ISPOR at https://tools.ispor.org/peguidelines/.12 As of the November 27, 2017, this comprised guidelines from 41 countries or regions, of which 33 were considered in the original study: Australia, Austria, Baltic States (Latvia, Lithuania, Estonia), Belgium, Brazil, Canada, Colombia, China, Croatia, Cuba, Czech Republic, Denmark, Egypt, England and Wales, the European Union, Finland, France, Germany, Hungary, Ireland, Israel, Italy, Japan, Malaysia, Mexico, New Zealand, The Netherlands, Norway, Poland, Portugal, Russian Federation, Scotland, Slovenia, South Africa, South Korea, Spain, Spain (Catalonia region), Sweden, Thailand, Taiwan, and the United States.13–54
For all documents included in the original study, and all documents identified in the ISPOR repository, the corresponding national health authority website was searched to ensure the document was the most recent available version, or to update with a more recently issued version. Each document was reviewed for its eligibility for this study. Documents were required to be either an HTA-related guideline or methodological review issued by an HTA body, to have provided sufficient documentation pertaining to NMA conduct to enable comparison, and to be available in English, Cantonese, or Mandarin.
Comparison of National Guidelines Extraction checklists were developed to summarize recommendations in the identified national guidelines, spanning the full breadth of NMA conduct, from identifying the evidence base to the methodological and statistical aspects of the quantitative evidence synthesis. The checklists were separated into the following
1180
VALUE IN HEALTH
OCTOBER 2019
Table 1. Recommendations regarding the conduct and reporting of literature review.
Design, conduct, and reporting of trial searches
Australia
Report date(s) search conducted
Specify search date span
Specify PICO criteria
Belgium
Justify exclusion criteria
Describe search terms and relationship
Present strategy in blocks by indication, intervention, study type, etc.
Perform supplementary searches
Present PRISMA diagram
Canada
Follow CRD systematic review procedures
China Ireland England & Wales
EUnetHTA France Germany
Scotland
South Africa
South Korea
Spain
Thailand
CENTRAL indicates Cochrane Central Register of Controlled Trials; CINAHL, Cumulative Index to Nursing and Allied Health Literature; CRD, Centre for Reviews and Dissemination; EED, Economic Evaluation Database; EUnetHTA, European Network of Health Technology Assessment; HEED, Health Economic Evaluation Database; ICTRP, International Clinical Trials Registry Platform; NHS, National Health Service; PICO, Patient-Intervention-Comparator-Outcome; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses. * eg, Australia New Zealand Clinical Trials Registry (formerly the Australia Clinical Trials Registry); Arzeneitmittel-information system; Pharmnet.BUND Klinische Prüfung. † eg, PsycINFO (psychology and psychiatry), MANTIS (osteopathy and chiropractic).
conceptual subgroups: (1) conduct and reporting of literature search, (2) study selection, (3) reporting and assessment of included studies, and (4) NMA. The review checklists were created through an iterative process designed to capture all recommendations and requirements. Where a national guideline made a recommendation, the associated item was added to the checklist unless already present. Uncertainty as to the interpretation of a recommendation, and whether concept duplication had occurred, was resolved through discussion with a second and, if required, third reviewer. Although time-consuming and broad in scope, this allowed us to capture every recommendation without judgment as to their methodological worth. Each checklist was completed for each of the guidelines by 2 separate reviewers. A final review of the guidelines was conducted, and any additional items required were added to the checklists where applicable. Finally, the checklists were compared across reviewers and any discrepancies were resolved through discussion with a third reviewer. Throughout the assessments, we did not differentiate between a recommendation and a requirement because the language used within the assessed documentation made this difficult to discern.
Results Of the 41 documents assessed, only guidelines documents from Australia, Belgium, Canada, China, England and Wales, France, Germany, Ireland, Scotland, South Africa, South Korea, Spain, and
Thailand made recommendations as to the conduct of NMAs.13–25 These guidelines, along with a guidance document issued by the European Network for Health Technology Assessment,26 comprise recommendations from 14 jurisdictions across Europe, Asia, North and South America, and Africa. Of the excluded guidelines, 10 documents mentioned the use of NMA but did not provide any recommendations as to the conduct of the analysis; these were issued by agencies in Croatia, Egypt, Hungary, Japan, The Netherlands, New Zealand, Norway, Poland, Sweden, and the United States. Nine documents were excluded because they made no mention of NMA techniques; these were issued by agencies in Austria, Denmark, Baltic States (Estonia, Latvia, and Lithuania), Finland, Israel, Italy, Colombia, Malaysia, and Taiwan. A further 7 documents were excluded because they were presented in a language other than English, Mandarin, or Cantonese (Fig 1, Appendix Table 1, see Supplemental Materials found at https://doi. org/10.1016/j.jval.2019.05.013).
Conduct and Reporting of Literature Search Table 1 summarizes the recommendations provided in the guidelines and methods documents relating to how the search for evidence was executed and reported. Of the 14 HTA bodies whose guideline documents were included in the review, 12 address some aspect of the literature search; neither China16 nor Ireland19 provide such guidance. Regarding the design, conduct, and reporting of the searches, there is an overall focus on repeatability; presenting the search
POLICY PERSPECTIVE
1181
Table 1. Continued Specified databases Medline
Embase
Cochrane (CENTRAL)
Cochrane (Systematic reviews)
Clinicaltrials. gov
Clinical trial register of EMA
Jurisdictionspecific databases*
CINAHL
ICTRP search portal
NHS CRD
Subject-specific databases or registers†
Manufacturers internal databases
Table 2. Study selection. Selection process
Trials include Maintained log of trials deemed ineligible
Trials form a single connected network of interventions
Requirement for homogeneity of prognostic severity
Predefined inclusion/ exclusion criteria process
Justified selection criteria
Australia
Belgium
Selected by 2 independent reviewers
England & Wales
France
Germany
Scotland
South Africa
Homogeneity between trials
EUnetHTA
Blinded outcome assessment
China Ireland
Blinded trial (participants/ personnel)
Canada
Randomized allocation*
South Korea Spain
Thailand EUnetHTA indicates European Network of Health Technology Assessment. *Criteria for randomized controlled trials.
1182
VALUE IN HEALTH
OCTOBER 2019
Table 3. Reporting and assessment of included studies. Reporting and assessment of clinical trials Describe the design and methodology according to CONSORT guidelines
Assessment of variance between trial protocol and standard practice
Assess adequacy of blinding
Comparison Implementation Assessment Describe Investigator of of ITT of time conflicts of rates of difference horizon interest dropout/ in baseline reported loss to risk and follow-up placebo response
Report of subgroup analysis
Treatment effect modifiers identified prior to comparing study results
Assessment of publication bias and/or funnel plot
Australia
Belgium
Canada
China Ireland
England & Wales
EUnetHTA
France Germany
Scotland
South Africa
South Korea
Spain
Thailand
CONSORT indicates Consolidated Standards of Reporting Trials; EUnetHTA indicates European Network of Health Technology Assessment; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; ITT, intent-to-treat.
Table 4. Conduct of network meta-analysis.
Australia
Description of patient & treatment characteristics
Describe & justify statistical method; Bayesian or Frequentist.
If Bayesian, describe prior distribution, sensitivity to priors, and assess convergence.
Present Individual study results and/or Forest plot
Identify and report influential studies using metrics (residuals, Cooks distance)
Graphically summarize rank probabilities
Description of relative-effect estimate
Include rationale for, and description of, sensitivity analyses
Description of different findings with sensitivity / scenario analysis
Belgium Canada
China
Ireland England & Wales
EUnetHTA
France
Germany
Scotland
South Africa
South Korea
Thailand
pain
POLICY PERSPECTIVE
strategies systematically; the use of specified patient, intervention, comparator, outcome criteria; and conducting supplemental searches including manual checking of the references in retrieved papers and gray literature searches. Only the Belgian guidelines specify that the search should comply with best-practice guidelines from a specific organization (specifically, the Centre for Reviews and Dissemination).55 Across the documentation, Australia, Belgium, Germany, and Scotland provide the largest number of specific requirements for the conduct of trial searches, and all 4 specify the following 2 requirements: (1) that search terms, such as medical subject heading identifiers, and other keywords used to search electronic databases should be described; and (2) that such descriptors are grouped into related blocks in the presentation of the search strategy.13,14,19,21 For the bibliographic databases and registries to be searched (Table 1), only half of the HTA bodies provide specific recommendations. Of those that do provide guidance, searches of Medline and EMBASE are required by all of the 7, and search of the Cochrane Central Registry of Controlled Trials is specified by 5 of these 7. Many guideline documents require additional searching in jurisdiction-specific clinical trial databases. The German guidelines require that searches include the German-language Arzeneitmittel-informations system and PharmnetBUND Klinische Prüfung, and the national guideline document issued by the Australian Pharmaceutical Benefits Advisory Committee requires that the Australia New Zealand Clinical Trials Registry form part of the search strategy.13,19 Guidelines issued by Australia, England and Wales, Germany, and Scotland require that company-specific databases be searched and results presented in the submission, although the guideline documents contain no indication of how the validity of such a search would be confirmed.13,17,19,21
1183
considerable biasing of results may occur through the selection of trials with varying levels of quality. Most of the guideline documents specify that inclusion or exclusion criteria for trials in the NMA should be defined before the selection process, and that selection criteria be justified in the submission (Table 2). In addition, 6 of the 14 guideline documents require that a record of trials that are deemed ineligible is presented alongside the trials that meet selection criteria, an important step in reducing the risk of selection bias and increasing the level of transparency. Nine of the 14 guideline documents found it necessary to explicitly state that random allocation to study arms was a requirement for inclusion in NMAs, and 6 documents specified that trials should be double-blinded (Table 2).
Reporting and Assessment of Included Studies Elements relating to the transparent reporting and critical appraisal of studies contributing to NMAs are summarized in Table 3, capturing the recommendations relating to this aspect of evidence synthesis provided by 12 of the 14 HTA bodies included in the review. There was consistency across the guideline documents, with most requiring an assessment of the level of bias within studies, an assessment of between-study differences in baseline risk and placebo response, and a description of the time horizon for the studies. Only Ireland and Germany specified that design and methodology were described according to Consolidated Standard of Reporting Trials guidelines.56 The requirement to provide a report of subgroup analysis was the most consistently applied item, and 5 guidelines required the use of a funnel plot to assess bias.
Conduct of NMA Study Selection The selection of the studies that will form the network of trials is a key part of the conduct of an NMA, and a step where
Table 4 describes the required statistical techniques, processes, and presentation of the NMA. No single aspect of the conduct of NMA is universally addressed by all HTA bodies. The most
Table 4. Continued MetaNaïve regression comparisons of point estimates or active arms are prohibited
Include analysis of the hypothesis of consistency and/or transitivity assumption
Assess the results for each common reference across trials for any important differences
Perform consistency check between direct and indirect evidence
Assess homogeneity of direct comparisons
Assess heterogeneity with Cochrane Q, I2
Compare Random Include the results effects diagram from the model of simple indirect network comparison structure method (Buchers difference)
Assess level of bias within included studies
Provide annotated software code
1184
VALUE IN HEALTH
frequently required features of NMA included the presentation of a random effects model, the justification of using either frequentist or Bayesian statistics, the assessment of homogeneity of direct comparisons, and the use of sensitivity analysis. None of the guideline documents stipulated what form sensitivity analyses should take, but 4 of the HTA bodies require a rationale for sensitivity analyses, 6 countries subsequently required a description of the different findings, and 4 needed a relative effect estimate. Most HTA bodies call for a consistency check between direct and indirect evidence, with the Australian guidelines suggesting the use of Bucher’s method, indicating that indirect evidence should be considered alongside direct evidence and the National Institute for Health and Care Excellence Decision Support Unit documentation providing details on the use of inconsistency models. The National Institute for Health and Care Excellence Decision Support Unit documentation provides suggestions on how to deal with inconsistency, if detected, suggesting re-evaluation of the choice of scale, effect modification, and similarity of the contributing randomized controlled trials. The documentation provides questions for critical evaluation of the source and impact of inconsistency but acknowledges that it may also be due to chance. None of the other national guidelines indicated that differences between direct and indirect evidence would render indirect evidence inadmissible. England and Wales was the only country to require the inclusion of code written for analysis and the statistical software packages used.17 Almost half the countries explicitly prohibited the use of naïve indirect comparisons, where the results of individual arms of different trials are compared as though they were from the same trial. Appendix Table 2 (see Supplemental Materials found at https://doi.org/10.1016/j.jval.2019.05.013) shows the various ways of presenting the results of an NMA and choices of scale for analysis. The various national guideline documents appear to take 2 approaches to the issue of data scales in the presentation of results: either to present a wide-ranging set of scales or to not present any requirements at all and leave the choice of scale the judgment of those conducting the NMA. Ireland requires that both absolute and relative effect scales are presented, but the specific choice of scale (eg, odds ratio, relative risk, or hazard ratio) is not addressed.
Discussion This study builds on a previous comparison of multiple national guideline documents.11 Using a similar study framework, the limited scope of the original study has been expanded to include additional countries and to include guidelines that have been updated. Compliance with the PRISMA statement is highly variable among the component guidelines of the national guidelines superset, with guidance on systematic review being outside the scope of countries such as Canada, China, and South Korea. Nevertheless, when considered as a compiled super-set of requirements, the national guidelines compare favorably to the PRISMA extension statement on the reporting of NMA. Of the domains on the PRISMA checklist related to conduct of study search, selection, and conduct of NMA, only items related to description of data extraction forms and reporting of methods used to explore the geometry of the network are lacking from the national guidelines super-set. The lack of requirements to assess biases implicit in network geometries is a notable omission because the PRISMA statement includes 3 items on its 27-domain checklist about the reporting of network structures and testing of alternate geometries.
OCTOBER 2019
The criteria for conducting NMA can broadly be categorized into the following: assessments and analyses to test assumptions required for a NMA, presentation and reporting of results, and justification for modeling choices. Where companies have limited time to conduct analyses, the criteria expected to have the largest impact on the results and conclusions for the NMA would be those assessing the assumptions of a NMA and, more specifically, the assessment of heterogeneity between studies. An example of a criterion that may not be impactful is requirement to test the sensitivity of the model results to the choice of prior for a Bayesian NMA. It may be time-consuming to perform this sensitivity check, and typically for models that converge well with sufficient number of studies and data points, it could be argued that this may not be necessary. Some of the criteria would not be expected to affect the conduct of an NMA and therefore may not be as necessary to meet. For instance, it may not be necessary to graphically summarize the rank probabilities, which are only required for England and Wales, and Germany. Nevertheless, considering that the ranking probabilities can easily be obtained from software such as WinBUGs (commonly used for Bayesian NMAs), exclusion of this step is unlikely to generate efficiencies. In addition, testing for consistency is typically conducted as a last step of the NMA process. Many published NMAs do not report the output of such assessment, although this is required for 8 of the 14 countries. Inconsistency in NMA results are usually explainable based on heterogeneity identified between studies. Therefore, if heterogeneity is appropriately assessed and adjusted for where feasible, this should lead to fewer inconsistencies being observed or at least the ability to explain some of the inconsistencies identified. Our assessment is currently the only attempt to collect together the disparate requirements for the conduct of NMA that the various national jurisdictions expect of reimbursement submissions. For the first time, this included countries from outside the Europe and commonwealth nations. China, South Korea, and Thailand all indicated their respective requirements for conduct of NMA in sufficient detail to allow them to be included.16,22,23,25 The limitations of this study arise from 3 sources: access to the source documents, limitations in the source documents themselves, and weaknesses in the review process. A limitation in accessing national guidelines is the lack of a searchable database. This review searched documents from a variety of sources, all of which were based online; however, the lack of a definitive repository for national guidelines means that the authors cannot be certain that all eligible guideline documents were captured by our search. Furthermore, online repositories that appeared comprehensive often hosted out-of-date versions of guideline documents or linked to pages that were no longer available, highlighting the difficulties involved in maintaining an online collection of living documents in the digital era. A further limitation is the effect of language restriction on the inclusion of guideline documents. Guidelines from Brazil, Iran, Mexico, and Cuba were identified but could not be appraised owing to lack of English language translations. The limitations of the guideline documents themselves were a lack of a shared vocabulary of nontechnical terms. Whilst techniques such as funnel plot or use of Cook’s distance are unambiguous and language-independent, nontechnical requirements may be interpreted subjectively and placed in the subcategory of the checklist where they best fit. The compounding effects of our subjective interpretations may be that readings of the guideline documents became informed by the guideline checklist tables themselves; however, our iterative, consensus-building approach was designed to minimize such bias.
POLICY PERSPECTIVE
Further work should focus on the requirements from emerging markets in South America, Africa, and South Asia and documentation relating to emerging techniques in evidence synthesis such as matching adjusted indirect comparisons, simulated treatment comparisons, and methods being used in the synthesis of nonrandomized evidence. This study expands the range of previous work that reviewed the guidelines for the use of indirect evidence from multiple national jurisdictions. These updated aggregate guidelines do not include requirements that are mutually prohibitive. It is possible to perform a single NMA for submission to a variety of national jurisdictions.
17.
Acknowledgments
21.
The authors have no other financial relationships to disclose.
18.
19.
20.
22.
Supplemental Material 23.
Supplementary data associated with this article can be found in the online version at https://doi.org/10.1016/j.jval.2019.05.013.
REFERENCES 1. 2. 3.
4.
5. 6.
7.
8.
9.
10.
11. 12.
13.
14.
15.
16.
Ioannidis J. Indirect comparisons: the MESH and MESS of clinical trials. Lancet. 2006;368:1470–1472. Caldwell DM, Ades AE, Higgins JP. Simultaneous comparison of multiple treatments: combining direct and indirect evidence. BMJ. 2005;331:897–900. Bucher HC, Guyatt GH, Griffith LE, et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol. 1997;50:683–691. Song F, Altman DG, Glenny AM, et al. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses. BMJ. 2003;326:472. Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Stat Med. 2004;23:3105–3124. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:2012. Hutton B, Salanti G, Caldwell DM, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med. 2015;162(11):777–784. Hoaglin DC, Hawkins N, Jansen JP, et al. Conducting indirect-treatmentcomparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health. 2011;14:429–437. Jansen JP, Fleurence R, Devine B, et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health. 2011;14:417–428. Jansen JP, Trikalinos T, Cappelleri JC, et al. Indirect treatment comparison/ network meta-analysis study questionnaire to assess relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17:157–173. Laws A, Kendall R, Hawkins N. A comparison of national guidelines for network meta-analysis. Value Health. 2014;17:642–654. International Society for Pharmacoeconomics and Outcomes Research. Pharmacoeconomics guidelines around the world. 2012. https://tools.ispor. org/peguidelines/. Accessed November 27, 2017. Pharmaceutical Benefits Advisory Committee. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee, version 5.0. 2016. https://pbac.pbs.gov.au/. Accessed November 27, 2017. Cleemput I, Neyt M, van de Sande S, Thiry N. Belgian guidelines for economic evaluations and budget impact analyses: second edition. Health Technology Assessment (HTA). Belgian Health Care Knowledge Centre. https://kce.fgov. be/sites/default/files/atoms/files/KCE_183_economic_evaluations_second_ed ition_Report_update.pdf. Accessed November 27, 2017. Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada – 4th edition. https://www.cadth.ca/dv/guidelines-economic-evaluation-health-technologi es-canada-4th-edition. Accessed November 27, 2017. China Center for Health Economics Research. China guidelines for pharmacoeconomic evaluation. 2011. https://tools.ispor.org/PEguidelines/ source/China-Guidelines-for-Pharmacoeconomic-Evaluations_2011_Chinese. pdf. Accessed November 27, 2017.
24. 25.
26.
27.
28. 29.
30.
31.
32. 33.
34.
35.
36.
37.
38.
39.
1185
National Institute for Health and Care Excellence. Guide to the methods of technology appraisal 2013. https://www.nice.org.uk/process/pmg9/resourc es/guide-to-the-methods-of-technology-appraisal-2013-pdf-2007975843781. Accessed November 27, 2017. Haute Autorite de Sante. Choices in Methods for Economic Evaluation. Department of Economic and Public Health, ed. Saint-Denis La Plaine, France; 2012. https://www.has-sante.fr/upload/docs/application/pdf/2012-1 0/choices_in_methods_for_economic_evaluation.pdf. Accessed November 27, 2017. Institute for Quality and Efficiency in Health Care. General methods (version 4.2) of 22 April 2015. Health Information and Quality Authority. Guidelines for the economic evaluation of health technologies in Ireland 2014. https://www.hiqa.ie/sites/default/files/2017-01/Revised_Eco nomic_Guidelines_posted_100714.pdf. Accessed November 27, 2017. South Africa Department of Health. The guidelines for pharmacoeconomic evaluations of medicines and scheduled substances. 2013. http://www.gpw online.co.za/Gazettes/Gazettes/36118_1-2_Health.pdf. Accessed November 27, 2017. Scottish Medicines Consortium. Guidance to Manufacturers for Completion of New Product Assessment Form (NPAF) 2017.22. https://www.scottish medicines.org.uk/making-a-submission/. Accessed November 27, 2017. National Evidence-based Healthcare Collaborating Agency (NECA). The Application of Bayesian Meta-Analysis in HTA: The Experience from Korea. 2014. https://www.neca.re.kr/report_download.do?seq=38&key=1&cms_menu_seq= 141. Accessed November 27, 2017. Bae S, Lee S, Bae EY, et al. Korean guidelines for pharmacoeconomic evaluation (second and updated version): consensus and compromise. Pharmacoeconomics. 2013;31:257–267. Ortega AF, Fraga MD, Alegre-del-Ray EJ, et al. A checklist for critical appraisal of indirect comparisons. Int J Clin Pract. 2014;68(10):1181–1189. Chaiyakunapruk N, Saokaew S, Sruamsiri R, et al. Systematic review and network meta-analysis in health technology assessment. J Med Assoc Thai. 2014;97(suppl 5):S33–42. European Network of Health Technology Assessment (EUnetHTA). EUnetHTAJA1 WP5 - comparators & comparisons: direct and indirect comparisons. 2015. https://www.eunethta.eu/wp-content/uploads/2018/01/ Comparators-Comparisons-Direct-and-indirect-comparisons_AmendedJA1-Guideline_Final-Nov-2015.pdf. Accessed November 27, 2017. Agency for Quality and Accreditation in Health Care Croatia. The Croatian Guideline for Health Technology Assessment Process and Reporting. 2011. http:// aaz.hr/sites/default/files/hrvatske_smjernice_za_procjenu_zdravstvenih_ tehnologija.pdf. Accessed November 27, 2017. Elsisi GH, Kalo Z, Eldessouki R, et al. Recommendations for reporting pharmacoeconomic evaluations in Egypt. Value Health Reg Issues. 2013;2(2):319–327. Ministry of Human Capacities - State Secretariat for Health. Professional healthcare guideline on the methodology of health technology assessment. Gyogyszereink. 2017;67(1). Central Social Insurance Medical Council. Guideline for preparing cost-effectiveness evaluation to the Central Social Insurance Medical Council. 2016. https://tools.ispor.org/PEguidelines/source/Japanese_PE_Guideline.pdf. Accessed November 27, 2017. Ijzerman MJ, Al MJ, de Boer A, et al. Guideline for economic evaluations in healthcare. Zorginstituut Nederland. 2016. https://tools.ispor.org/PEguide lines/source/Netherlands_Guideline_for_economic_evaluations_in_healthcare .pdf. Accessed November 27, 2017. Pharmaceuticals Management Agency (PHARMAC). Guidelines for funding applications to PHARMAC; 2017. Legemiddelverk Statens. Guidelines on how to conduct pharmaco economic analyses. 2012. https://tools.ispor.org/PEguidelines/source/ Norwegian_guidelines2012.pdf. Accessed November 27, 2017. Agencja Oceny Technologii Medyczncyh. Guidelines for Conducting Health Technology Assessment (HTA). AOTM; 2009. https://tools.ispor.org/ PEguidelines/countrydet.asp?c=16&t=2. Accessed November 27, 2017. Tandvårds-och läkemedelsförmånsverket. Handbok till Tandvårds- och läkemedelsförmånsverkets föreskrifter (TLVFS 2011:3) om ansökan om pris och subvention för förbrukningsartiklar Sweden. 2011. https://www.tlv.se/ download/18.467926b615d084471ac3390c/1510316399612/Handbok-TLVFS2 011-3-subvention-forbrukningsartiklar-sept-2011.pdf. Accessed November 27, 2017. Academy of Managed Care Pharmacy (AMCP). The ACMP Format for Formulary Submissions Version 4.0. Alexandria, VA: Academy of Managed Care Pharmacy; 2016. https://www.amcp.org/sites/default/files/2019-03/AMCPFormat-V4.pdf. Accessed November 27, 2017. Walter E, Zehetmayr S. Guidelines on Health Economic Evaluation. Vienna, Austria: Institut fur Pharmaokonomische Forschung; 2006. http://www.ipfac.at/fileadmin/template/PDF%20alt/Konsens_Guidelines_en.pdf. Accessed November 27, 2017. Kristensen FS H, Andersen SE, Andersen S, Bonnevie B. Health Technology Assessment Handbook. Denmark: Danish Centre for Health Technology Assessment, National Board of Health; 2007. https://www.sst.dk/w/media/ ECAAC5AA1D6943BEAC96907E03023E22.ashx. Accessed November 27, 2017. Behmane D, Lambot K, Irs A, et al. Baltic Guideline for Economic Evaluation of Pharmaceuticals. Medicines Pricing & Reimbursement Agency, Latvia; Ministry of Health, Lithuania; Health Insurance Fund, Estonia; Ministry of Social
1186
40.
41.
42.
43.
44.
45.
46.
47.
VALUE IN HEALTH
Affairs, Estonia; 2002. https://tools.ispor.org/PEguidelines/source/Baltic-PEguideline.pdf. Accessed November 27, 2017. Lääkkeiden Hintalautakunta. Preparing a Health Economic Evaluation to Be Attached to the Application for Reimbursement Status and Wholesale Price for a Medicinal Product: Application Instructions. Finland: Lääkkeiden Hintalautakunta; 2017. http://www.hila.fi/c/document_library/get_file?folderId=1133 981&name=DLFE-9844.pdf. Accessed November 27, 2017. Ministry of Health (Israel) - Pharmaceutical Administration. Guidelines for the Submission of a Request to Include a Pharmaceutical Product in the National List of Health Services - Appendix D: Instructions for Performing an Economic Assessment. Israel: Ministry of Health; 2010. https://tools.ispor.org/PEguide lines/source/Israel-Guidelines-for-submission_2010.pdf. Accessed November 27, 2017. Capri S, Ceci A, Terranova L, et al. Guidelines for economic evaluations in Italy: recommendations from the Italian group of pharmacoeconomic studies. Drug Inf J. 2001;35(1):189–201. Faria R, Duarte A, McKenna C, Revill P, Cary M, Byford S. Guidelines for the economic evaluation of healthcare technologies in Colombia: technical support documents. Bogota, Colombia: Instituto de Evaluacion Tecnologica en Salud; 2014. http://www.iets.org.co/Archivos/67/Documentos-tecnicosespanol.pdf. Accessed November 27, 2017. Ministry of Health Malaysia, Pharmaceutical Services Division - MIH Technical Working Group. Pharmacoeconomic Guideline for Malaysia. 2012. https://www.pharmacy.gov.my/v2/en/documents/pharmacoeconomic-guide line-malaysia.html. Accessed November 27, 2017. Taiwan Society for Pharmacoeconomics and Outcomes Research (TaSPOR). Guidelines of Methodological Standards for Pharmacoeconomic Evaluations in Taiwan (version 1.0). Taiwan. 2006. https://tools.ispor.org/PEguidelines/ source/2006_PEG_EN_2009.pdf. Accessed November 27, 2017. Ministerio da Saude. Diretrizes Metodologicas. Diretriz de Avaliacao Economica. 2nd ed. Brazil: Ministerio da Saude; 2014. http://bvsms.saude.gov.br/bvs/ publicacoes/diretrizes_metodologicas_diretriz_avaliacao_economica.pdf. Accessed November 27, 2017. Ministry of Public Health. Methodological Guidelines for Health Economic Evaluation. Cuba: Ministry of Public Health; 2003. http://scielo.sld.cu/scielo.php? script=sci_arttext&pid=S0864-34662004000100005. Accessed November 27, 2017.
OCTOBER 2019
48.
49.
50.
51.
52.
53.
54.
55.
56.
State Institute for Drug Control (SUKL). Cost-effectiveness Guidelines. Czech Republic: State Institute for Drug Control (SUKL); 2017. http://www.sukl.cz/ file/85481_1_1. Accessed November 27, 2017. National Institute of Health Research of the Islamic Republic of Iran. Guidelines for the Formation of HTA Proposal. Iran: National Institute of Health Research of the Islamic Republic of Iran; 2017. http://nihr. tums.ac.ir/upfiles/documents/307113275.pdf. Accessed November 27, 2017. Consejo de Salubridad General. Economic Assessment Study Guideline for Updating the National Formulary in Mexico. Mexico: Consejo de Salubridad General; 2015. http://www.csg.gob.mx/descargas/pdf/priorizacion/cuadrobasico/guias/insumos_salud/GEI_2017_Diciembre.pdf. Accessed November 27, 2017. Center for Healthcare Quality Assessment and Control of the Ministry of Health of the Russian Federation. Guidelines for Conducting a Comparative Clinical and Economic Evaluation of Drugs. Russia: Center for Healthcare Quality Assessment and Control of the Ministry of Health of the Russian Federation; 2016. https://rosmedex.ru/wp-content/uploads/2016/12/MR-KE% 60I-23.12.2016.pdf. Accessed November 27, 2017. Mora-Ripoli R, Gilabert-Perramon A, Oliva-Moreno J, Puig-Junoy J. Guidance for economic evaluation and budget impact analysis for pharmaceuticals in Catalonia (Spain). Value Health. 2014;17(7):A447. Health Insurance Institute of Slovenia. Rules on Classification of Medicines on the List, Accepted by the Health Insurance Institute of Slovenia. 2013. https:// www.farmaforum.si/sites/43/files/files/Pravilniki/Pravilniki%20ENG/Rules%20of% 20reimbursment.pdf. Accessed November 27, 2017. Alves da Silva E, Gouveia Pinto C, Sampaio C, Pereira JA, Drummond M, Trindade R. Guidelines for Economic Drug Evaluation Studies. Lisbon, Portugal: Autoridade Nacional do Medicamento e Produtos de Saude; 1998. http://www.infarmed.pt/portal/page/portal/INFARMED/ENGLISH/PCAEC04_ vering.pdf. Accessed November 27, 2017. University of York. Centre for Reviews and Dissemination. Systematic Reviews. CRD’s Guidance for Undertaking Reviews in Health Care. York, England: University of York; 2009. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.