- Email: [email protected]
How to Use a Clinical Practice Guideline
Users’ Guide to the Urological Literature
How to Use a Clinical Practice Guideline Philipp Dahm,*,† Lawrence L. Yeung,† Michele Gallucci, Giuseppe Simone and Holger J. Schünemann From the Department of Urology, College of Medicine, University of Florida, Gainesville, Florida (PD, LLY), Department of Urology (MG) and Epidemiology (HJS), Italian National Cancer Institute Regina Elena, Rome, and Department of Urology, Magna Graecia University of Catanzaro, Catanzaro (GS), Italy
Abbreviations and Acronyms AUA ⫽ American Urological Association Submitted for publication August 5, 2008. Nothing to disclose. * Correspondence: Department of Urology, University of Florida College of Medicine, Health Science Center, Box 100247, Room M2-204, Gainesville, Florida 32610-0247 (telephone: 352273-6815; FAX: 352-273-7515; e-mail: p.dahm@ urology.ufl.edu). † Equal study contribution.
Editor’s Note: This article is the second of 5 published in this issue for which category 1 CME credits can be earned. Instructions for obtaining credits are given with the questions on pages 928 and 929.
Purpose: Leading organizations increasingly recognize clinical practice guidelines as an important approach in promoting an evidence-based clinical practice of urology. In light of their considerable clinical, economic and medicolegal impact, guideline users should be confident that the guidelines were rigorously developed and address relevant patient questions. In this article we outline a practical approach for critically appraising a clinical practice guideline. Materials and Methods: We outline a 3-step approach to the assessment of a clinical practice guideline that answers the questions of whether the recommendations are valid, what the recommendations are and whether they will help in the treatment of an individual patient. Results: To determine the adequacy of a clinical practice guideline, urologists should carefully review the rigor of the development process and its content. Important questions that relate to the validity of a guideline are whether, for specific questions, all important management options and outcomes were considered, and whether there was an explicit and sensible process to identify, select and combine all relevant evidence. Clinical practice guidelines should formally grade the quality of the available evidence for a given clinical question and outline a formal process of how the recommendations were derived. Value judgments made in the guideline development process about the relative importance of the potential benefits and harms of a given health care intervention should be made transparent to the reader. The recommendations made should be practical and should address important clinical issues. Furthermore, their strength should be graded to reflect the underlying uncertainty about the evidence and the values applied in the guideline development process. Conclusions: The systematic approach presented in this article will allow urologists to critically appraise clinical practice guidelines. Determining the validity of the recommendations, understanding the recommendations and assessing their applicability to patients are 3 fundamental steps toward an evidence-based approach to using clinical practice guidelines. Key Words: practice guidelines as topic, evidence-based medicine
472
www.jurology.com
UROLOGISTS are continuously faced with the need to make or facilitate decision making for individual patients. Making treatment recommendations implicitly or explicitly involves formulating a clinical question, identifying management
options and outcomes, collecting and summarizing evidence, and applying value judgments and preferences to arrive at an optimal treatment plan. In an attempt to facilitate clinical decision making and promote the delivery of
0022-5347/09/1812-0472/0 THE JOURNAL OF UROLOGY® Copyright © 2009 by AMERICAN UROLOGICAL ASSOCIATION
Vol. 181, 472-479, February 2009 Printed in U.S.A. DOI:10.1016/j.juro.2008.10.041
HOW TO USE CLINICAL PRACTICE GUIDELINES
consistent, high quality care to all patients, clinical practice guidelines are being developed by various local, regional and international organizations. Clinical practice guidelines seek to provide explicit recommendations for the treatment of typical index patients based on the best available research evidence, while at the same time assessing the balance between desirable and undesirable effects, the patient’s presumed values and preferences, and the potential economic impact of different treatment options. As such, the Institute of Medicine defines them as “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.”1 Unfortunately the methodological quality of clinical practice guidelines developed by different organizations varies considerably, thereby impacting the confidence that we can place in their recommendations.1– 4 Therefore, it is important that urologists be familiar with the defining characteristics of clinical guidelines that deserve the label evidence-based, and that they have a conceptual framework to determine whether a guideline document is likely to be valid, to interpret the recommendations appropriately and to determine whether the recommendations are applicable to a particular patient. Thus, we introduce a systematic approach to critically evaluate a clinical practice guideline document before applying its recommendations to the care of an individual patient.
CLINICAL SCENARIO You are a community practice urologist and are previewing your list of clinic patients for the next day which includes the brother of a close friend. Your friend asked for a second opinion for this 61-year-old patient with newly diagnosed prostate cancer. Outside records have been faxed to your office. The patient has clinical T1c disease with Gleason score 3 ⫹ 3 ⫽ 6 adenocarcinoma of the prostate in 4 of 14 cores and a prostate specific antigen of 4.5 ng/ml. The urological history is negative for any significant urinary complaints but mentions poor erectile function. The medical history is notable for mild hypertension, hyperlipidemia and adult onset diabetes which is well controlled with an oral agent. Having recently attended a urological meeting where much emphasis was placed on the importance of evidence-based decision making consistent with clinical practice guidelines, you feel compelled to update your knowledge on the management of clinically localized prostate cancer. Therefore, you decide to retrieve the most recent clinical practice guidelines on prostate cancer to guide the treatment of this patient.
473
SEARCH OF THE MEDICAL LITERATURE From your reading of the “Users’ Guide to the Urological Literature” by Krupski et al on how to search the medical literature effectively, you recall the National Guideline Clearing House as a dedicated resource for evidence-based clinical practice guidelines.5,6 Using prostate cancer as single search term you find 65 guideline documents (accessed June 1, 2008). Considering recent advances in the understanding of the natural history of prostate cancer as well as new treatment approaches including active surveillance, you decide to narrow the search by applying filters for treatment (vs diagnosis or prognosis) and the last 5 years only (2004 to 2008), which yields 27 guidelines. Among these only 1 document with the title “Guidelines for the Management of Clinically Localized Prostate Cancer: 2007 Update” strikes you as truly applicable to your patient scenario.7,8 Delighted to have found a recent guideline by a leading urological association you decide to print a copy of this document and review it before your patient’s clinic visit.
ARE THE RECOMMENDATIONS VALID? Before using evidence provided by a clinical research study, urologists should ask whether the results of the study are likely to be valid.9 The same principle applies to clinical practice guidelines.10 Guideline users should be confident that these recommendations are the product of a rigorous evidence driven development process, that the recommendations address clinically relevant issues and are applicable to the patient for which guidance is being sought. To assess the methodological quality of clinical practice guidelines several validated instruments have been developed.11 Among them the Appraisal of Guidelines for Research and Evaluation (AGREE) collaboration instrument is one of the best known and most widely used.12 It distinguishes 6 domains of methodological quality and transparent reporting. 1) “Scope and purpose” deals with the overall aim of the guideline, the specific clinical questions and the target patient population. 2) “Stakeholder involvement” focuses on the extent to which the guideline represents the views of its intended users. 3) “Rigor of development” relates to the process used to gather and synthesize the evidence and the methods to formulate the recommendations, and to update them. 4) “Clarity and presentation” deals with the language and format of the guideline. 5) “Applicability” pertains to the likely organizational, behavioral and cost implications of applying the guideline. 6) “Editorial independence” addresses the independence of the recommendations and acknowledgment of possible conflict of interest from the guideline development group. A similar endeavor to establish a shared framework for guideline development, reporting and assessment
474
HOW TO USE CLINICAL PRACTICE GUIDELINES
has been put forward by COGS (the Conference on Guideline Standardization). This group, with representatives from 22 professional organizations, used a 2-stage modified Delphi process to develop standards for reporting clinical practice guidelines which resulted in an 18-item COGS checklist as a consensus standard for reporting.4,13 Also, the guideline content presented in the National Guideline Clearinghouse represents a standard for reporting imposed on organizations that want their guidelines included in that database. It is composed of 52 items including the topic scope, methodology, evidence supporting the recommendations, benefits/harms of implementing the recommendations, contraindications and implementation of the guideline among others. However, for the purpose of the clinical reader a more rapid assessment of study validity relies on a few key questions (Appendix 1). Were All Important Options and Outcomes Considered? Guidelines pertain to clinical decision making, and decisions involve choices and consequences.14 To understand why a particular practice is recommended you should consider whether the guideline developers included all reasonable practice options and important consequences. For example, when it comes to the treatment of patients with clinically localized prostate cancer you expect guideline developers to have considered radical surgery, radiation therapy, active surveillance and potentially cryotherapy as therapeutic options. Important patient relevant outcomes include long-term survival, quality of life, urinary and sexual function as well as information about the morbidity associated with different options. Ideally guideline developers should also present a hierarchy of the important patient outcomes that they considered, and distinguish between critical and noncritical outcomes for clinical decision making.15 This distinction should be made at an early stage of the guideline development process. As a cost conscious provider you should also look for economic considerations in the guideline document.16,17 When economic outcomes are taken into account it is important to realize which perspective was chosen, such as that of the patient, insurer or health care systems overall, and whether broader issues such as the consequences of time lost from work were included which can have a major impact on the final recommendations.14,18 Was an Explicit and Sensible Process Used to Identify, Select and Combine Evidence? Once patient relevant outcomes and interventions have been defined, the next task in decision making is to estimate the likelihood that a given outcome will occur. To do so guideline developers should define a series of specific questions.15 For example, one such focused clinical question would be, “In patients with clinically localized prostate cancer, how does treat-
ment with radical prostatectomy compare to active surveillance with regard to long-term survival?”. The structure of this question follows the same Population, Intervention, Comparison and Outcome (PICO) format introduced by Krupski et al.6 Guideline developers must then bring together all relevant evidence and combine it in an appropriate manner that avoids the introduction of bias. The criteria by which to judge whether guideline developers have done an adequate job in accumulating and synthesizing the evidence are similar to those that apply to systematic reviews.19 Guideline developers should specify a focused question, define appropriate evidence using explicit inclusion and exclusion criteria, conduct a comprehensive search and examine the validity of the results in a reproducible fashion. Guidelines should report what type of evidence was considered, report how it was selected, abstracted and combined, and make key data available for review. We will review important methodological criteria by which to assess the rigor of the guideline development process. Is There a Description of Individuals Involved in the Development Process? For clinical practice guidelines to provide meaningful recommendations it is important that explicit and implicit judgments made by a panel reflect the views of its intended users. The guideline panel is responsible for interpreting the published evidence with regard to their own knowledge and experience in their respective fields and, therefore, should have broad representation. Multispecialty panels involving members with a variety of backgrounds such as patients, primary care providers, specialists, nonphysician providers (eg nurses, physical therapists, patient advocates), research experts and health care policy makers are more likely to produce balanced guidelines than a panel composed of only urologists, for example. A panel with a varied background will provide a diversity of experience and philosophy on health care, thereby limiting potential bias in the development process. Were Conflicts of Interest of the Guideline Panel Members Considered? In light of the considerable implications that guideline recommendations for or against certain treatment modalities may have, the guideline panel should provide assurance to potential users that the development process was not influenced by special interest groups. Although it may be beneficial for the guideline panel to include clinical investigators who have served as consultants or principal investigators of industry sponsored trials, any potential conflicts of interest need to explicitly identified. In addition, there needs to be mechanisms in place to manage any conflicts of interest should they arise.
HOW TO USE CLINICAL PRACTICE GUIDELINES
Are the Sources of Evidence Used in the Guideline Development Identified? Clinical practice guidelines should be based on a comprehensive and systematic review of the entire body of evidence, published and unpublished, that is applicable to the clinical question of interest. Details of the strategy used to obtain the evidence should be explicitly stated including the search terms used, the sources used and the time frame of the literature that was incorporated into the clinical practice guideline. Criteria for including or excluding evidence identified by the search should also be provided. What Were the Methods Used to Interpret and Assess the Strength of Evidence? One of the guiding principles of evidence-based medicine is the concept of a hierarchy of evidence. It recognizes that certain study designs are more suitable than others to provide an accurate representation of the truth, ie be less biased. For questions regarding therapeutic effectiveness, randomized controlled trials provide the highest level of evidence. Ideally, these randomized controlled trials show similar findings (ie low heterogeneity) and are summarized in systematic reviews that contain meta-analyses. However, individual randomized controlled trials need to address certain methodological safeguards against bias to be considered valid. Major limitations that may bias the estimates of the treatment effect include failure to implement allocation concealment, lack of blinding especially if the outcomes are subjective and their assessment highly susceptible to bias, and large losses to followup. Additional issues include the failure to adhere to intent to treat analysis and the issue of stopping trials early for apparent benefit.20 –22 Therefore, guideline developers need to critically analyze the methodological quality of the studies on which they base their recommendations rather than relying on the study design alone, and downgrade the evidence that is derived from studies with serious methodological flaws. Were the Methods Used to Formulate Recommendations Described? Once the quality of evidence has been established, guideline developers need to integrate and synthesize the available evidence to generate recommendations that provide specific guidance for clinicians, patients and policy makers. The strength of recommendations should be graded to reflect the degree of certainty that the desirable effects of adherence to a recommendation outweigh the undesirable effects. Key considerations that determine the strength of a recommendation include the quality of the evidence, the balance between desirable and undesirable consequences of alternative management strategies, the degree of uncertainty or variability of patient values and preferences for which the guideline document is intended and finally, the
475
associated costs.15,23 The guideline process that is used to integrate these factors may vary by the methodological framework that is used, but needs to be transparent and reproducible. Specifically the judgments that panelists form during the guideline development process need to be explicit to the user. These judgments relate to the relative importance of certain outcomes compared to others and assumptions that are being made about patient preferences. Has the Guideline Been Subjected to Peer Review and Testing? Similar to any peer reviewed publication, clinical practice guidelines should be externally reviewed before their public dissemination. The purpose of this external review is to provide other individuals with vested interests and expertise in this area to scrutinize the rigor of the guideline development process, as well as the appropriateness and applicability of the recommendations being made. The external review should be considered an integral part of the guideline development process, follow clearly defined guidelines and be clearly documented. In addition, it is highly recommended to test the application of a clinical practice guideline in a sample of patients that is similar to the population to which the guidelines will ultimately apply. Pilot testing provides valuable insights into the practical applicability of a guideline document and helps to understand its potential downstream effects. Are Methods for Updating the Guideline Document Described? Clinical practice guidelines should be based on the current best available evidence. Since new research evidence is published on a daily basis, guidelines invariably become outdated unless there are mechanisms in place to keep them updated. The frequency of such updates would be dictated by how rapidly new research findings become available in a certain field. Therefore, guidelines should have a mechanism in place to ensure that the recommendations are routinely reviewed and do not become outdated. Validity Assessment of the Clinical Practice Guideline on Prostate Cancer The AUA 2007 guideline document on clinically localized prostate cancer appears to address all patient relevant outcomes except resource use.7,8 It describes the individuals who were involved in the guidelines development as a multidisciplinary team. The evidence source is clearly identified although limited to the English literature that was indexed by the National Library of Medicine. The process of data abstraction is outlined and several quality checks were included to assure accuracy. The data were then compiled and presented to the guideline panel for discussion. No level of evidence grading system was applied to studies considered by the guideline panel, and no
476
HOW TO USE CLINICAL PRACTICE GUIDELINES
formal process as to how the recommendations were formulated has been reported. Although not pilot tested, the AUA guidelines were peer reviewed. The guidelines document addresses how the guidelines will be disseminated. A mechanism for updating the guidelines appears in place. In summary, shortcomings of the AUA guidelines are the failure to consider economic implications, clearly define the quality of underlying evidence and describe a formal process by which the recommendations were formulated. Meanwhile, many other important validity criteria appear to be met. Therefore, we would conclude that recommendations are likely to be valid with the previously mentioned reservations and, thus, will review them using a similar systematic approach.
WHAT ARE THE RECOMMENDATIONS? Once you have determined that the guideline recommendations are likely to be valid, you should review the actual recommendations that are being made. Are Practical, Clinically Important Recommendations Made? Clinical practice guidelines are developed to assist clinicians, patients and health policy makers in clinical decision making. Therefore, guideline recommendations need to provide practical, unambiguous advice for clearly defined problems that arise in the treatment of patients.4 Shekelle et al evaluated the effect of different levels of specificity of recommendations on clinicians’ test ordering behavior using clinical vignettes.24 They found that clinicians receiving the nonspecific recommendations ordered fewer indicated tests for appropriate indications than physicians who received specific recommendations. The authors concluded that the clarity and clinical applicability of a guideline are important attributes that contribute to the effects of practice guidelines. How Strong are the Recommendations? The strength, grade, confidence or force of recommendations made by guideline developers reflects the extent to which we can be confident that the desirable effects of an intervention outweigh the undesirable effects.25 In addition, the strength of a recommendation reflects considerations about the quality of the evidence, the necessary resources, and the extent by which the values and preferences of patients regarding the importance of certain outcomes vary. Therefore, the grading of recommendations is an important way for guideline developers to convey this level of confidence to the user. While most guideline development groups now grade the quality of evidence and the strength of recommendations, a variety of different grading systems are being used, thereby making it difficult to compare recommendations by different or-
ganizations and possibly impeding guideline implementation.26,27 Summary of Recommendations Made by the AUA Practice Guidelines The treatment options in the 2007 AUA guidelines for localized prostate cancer address several practically important management decisions. At the same time many important issues in clinically localized prostate cancer that clinicians and patients face on a day-to-day basis remain unaddressed. For example, the guideline document fails to provide any guidance on issues relevant to urological surgeons such as when to choose one surgical approach over another or which patients should undergo lymph node dissection for clinically localized prostate cancer. The AUA guidelines graded the strength of its recommendations using a 3-tiered system of flexibility based on the strength of evidence and the panel assessment of patient needs and preferences as standards, recommendations and options which are intended to signal to users the level of confidence that should be placed into these recommendations (Appendix 2).28 Recommendations were labeled standard if the panel concluded that it should be followed by virtually all health care providers for virtually all patients. However, few of the recommendations were considered standards which are intended to provide strong guidance for clinical decision making. The guideline panel formulated a recommendation if the members thought it was appropriate not for all, but for a significant majority of patients. Recommendations were graded as option if the panel determined that the health outcomes were not sufficiently wellknown and no clear patient preferences were apparent. One major underlying issue for the failure to address other important clinical questions and the paucity of strong recommendations is clearly stated in the guideline document, and relates to a lack of high quality evidence for many important questions in prostate cancer.27 However, there are also considerable issues with the system used for grading the strength of the recommendation itself. A recommendation labeled as an option provides little guidance to the clinician at all and, therefore, could be eliminated. In addition, the distinction between standard (which should be applied to virtually all patients) and recommendation (applicable to a significant majority) is ambiguous. Finally, based on the definitions provided there is not a single situation in which clinicians and patients could not deviate from a standard or recommendation. Therefore, it appears that the ambiguity of the recommendations would limit their practical applicability in the clinical care of patients.
HOW TO USE CLINICAL PRACTICE GUIDELINES
WILL THE RECOMMENDATIONS HELP YOU IN CARING FOR YOUR PATIENTS? Having assessed the validity of the guideline development process and reviewed the actual recommendations being made you should next ask whether the clinical guideline will actually help you in treating your patients by considering 3 questions. Is the Primary Objective of the Guideline Consistent With Your Objectives? Clinical practice guidelines are being developed for different purposes and different target audiences. Most commonly we think of guidelines as instruments to assist physicians and patients with clinical decision making. However, guidelines are also being developed to evaluate physician practices for quality assurance purposes or to set limits with regard to physician choices for reimbursement. Prostate cancer guidelines may also target other types of health care providers such as radiation oncologists, medical oncologists and nurses. Therefore, you should ensure that the purpose of the guideline is consistent with the use you intend for it. Are the Recommendations Applicable to Your Patients? To be really useful guidelines should describe interventions or procedures well enough for their exact duplication. You must also determine whether your patient characteristics fit the intended target audience of the guideline. If your patient has a different prevalence of disease or risk factors, for instance, the guideline may not apply. Ideally in the face of important differences in patient or practice characteristics, guideline flexibility allows users to individualize recommendations or justify departure from the recommendations altogether. For example, the risk stratification of clinically localized prostate cancer into low, intermediate and high risk categories by the AUA guidelines is an example of differing recommendations for different prognostic groups. Finally, you should look for information about the values that were assigned explicitly or implicitly to alternative outcomes. If the assigned values differ sufficiently from patient preference, this discrepancy may impact the decision to implement a recommendation. Applicability of AUA Guidelines to the Patient in the Clinical Scenario A review of the AUA guidelines document confirms that its primary purpose is to guide provider and patient decision making in patients with clinically localized prostate cancer. Developed by the AUA, urologists are the main target audience and the guideline objectives appear to be consistent with your own. In addition, your patient appears to be
477
similar to those for whom the guideline was developed, and likely shares similar values and preferences. Therefore, you conclude that the AUA guideline document is likely applicable to your patient.
RESOLUTION OF CLINICAL SCENARIO After a thorough review of the 2007 AUA guidelines on clinically localized prostate cancer you feel well prepared to counsel this patient. A thorough review of patient history of present illness, medical history and review of systems confirms the information you were provided. Based on the available clinical information, you determine that the patient has a statistical life expectancy of well over 10 years and falls into the low risk category based on the D’Amico classification system and, therefore, is a good candidate for local treatment with curative intent.29 To decide whether you can and should apply the AUA guidelines to the care of this patient you once again consider the 3 central questions recommended by the Users’ Guide to the Urological Literature. With regard to the validity of the guidelines you determine that the guideline document is more likely to be valid than not. Key shortcomings were found in the lack of formal grading of the quality of evidence, and a transparent and well-defined process of how guideline developers formulated the recommendations. With regard to what the recommendations are, you found the system used to grade the strength of recommendations to be ambiguous and of limited practical value. In addition, several clinically important questions remained unaddressed, likely because the evidence to address them was lacking. Finally, with regard to the question of whether the guideline document will actually help you in caring for your patient, you found its objectives to be consistent with your own and the guideline document applicable to your patient. In summary, your critical review of the guideline has identified a series of issues which hinder its use in clinical decision making and limit its practical value. That said, the guideline document provides a comprehensive overview of available treatment options as well as their associated benefits and potential side effects. Also, many of the recommendations defined as a standard appear sensible and noncontroversial. Specifically you decide to follow the guideline recommendation to provide your patient with a comprehensive overview of available treatment options including active surveillance, radiotherapy and radical surgery. In addition, you dedicate a considerable amount of time exploring patient opinions and preferences with regard to the potential urinary, sexual and bowel adverse events as the result of treatment. You learn through this conver-
478
HOW TO USE CLINICAL PRACTICE GUIDELINES
sation that he is anxious about delaying curative treatment due to concerns about disease progression and, therefore, reluctant about active surveillance. He places little importance on erectile function due to his marital situation and not having been sexually active for many years. Finally he expresses serious concerns about the risk of bowel related side effects whereas the risk of some degree of stress urinary incontinence after surgery is less threatening to him. Nevertheless, you suggest he see a radiation oncologist for consideration of primary radiation treatment, to which he agrees. You also provide him with a copy of the patient guide for the management of localized prostate cancer which is published by the AUA as a supporting document for the full guidelines document. Several weeks later he returns to your office for further counseling at which time he indicates his decision to undergo radical prostatectomy. Surgery and postoperative course are uneventful, and the patient experiences a timely recovery of urinary function and no short-term complications. The patient is most grateful for the treatment he received, and specifically thanks you for the time and effort you spent counseling him to guide his decision making.
CONCLUSIONS Clinical practice guidelines are becoming increasingly relevant evidence-based resources to guide clinical decision making for urologists.30 This article provides a basic framework for the critical appraisal of clinical practice guidelines using the AUA guidelines for clinically localized prostate cancer as an example. Guideline users should ask whether the recommendations are valid, what the recommendations are and whether they can be applied to the care of individual patients. These 3 questions are independently important and should be answered affirmatively before the guidelines are applied to actual patient care. Guidelines that meet these criteria can be an excellent guide for individual clinical decision making when integrated with the individual values and preferences of the patient.23,31,32 Evidence-based clinical practice guidelines also provide the basis of quality of care measures which are currently under development and likely to impact every urology practice in the future.33,34 Therefore, guideline methodology is of paramount importance and the subject of considerable scrutiny. Recognizing the importance of a transparent and evidence-based guideline development process, and
following careful assessment of various methodological approaches, the AUA has recently announced a major overhaul of its guidelines development process. The revised process is expected to address many of the shortcomings highlighted in this article which include explicit grading of the quality of evidence and clearer definitions of the strengths of recommendations. Efforts to move towards a unified approach to guideline development by all urological associations appear increasingly important. A joint guideline development process would provide tremendous opportunities for international consensus building, cost reduction and ultimately improved quality of care for all urological patients.
APPENDIX 1 Users’ Guide to a Clinical Practice Guideline Are the recommendations valid? Were all important options and outcomes considered? Was an explicit and sensible process used to identify, select and combine evidence? Is there a description of individuals involved in the development process? Were the conflicts of interest of the guideline panel members accounted for? Are the sources of evidence used in the guideline development identified? What were the methods used to interpret and assess the strength of evidence? Were the methods used to formulate recommendations described? Has the guideline been subjected to peer review and testing? Are methods for updating the guideline document described? What are the recommendations? Are practical, clinically important recommendations made? How strong are the recommendations? Will the recommendations help you in caring for your patients? Is the primary objective of the guideline consistent with your objectives? Are the recommendations applicable to your patients?
APPENDIX 2 Three-Tiered System Used by the 2007 AUA Guidelines on Clinically Localized Prostate Cancer to Grade the Strength of Recommendations Standard
Recommendation
Option
A guideline statement is a standard if: (1) the health outcomes of the alternative interventions are sufficiently well-known to permit meaningful decisions, and (2) there is virtual unanimity about which intervention is preferred. A guideline statement is a recommendation if: (1) the health outcomes of the alternative interventions are sufficiently well-known to permit meaningful decisions, and (2) an appreciable but not unanimous majority agrees on which intervention is preferred. A guideline statement is an option if: (1) the health outcomes of the interventions are not sufficiently well-known to permit meaningful decisions, or (2) preferences are unknown or equivocal.
HOW TO USE CLINICAL PRACTICE GUIDELINES
479
REFERENCES 1. Shaneyfelt TM, Mayo-Smith MF and Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 1999; 281: 1900. 2. Guyatt G, Vist G, Falck-Ytter Y, Kunz R, Magrini N and Schunemann H: An emerging consensus on grading recommendations? ACP J Club 2006; 144: A8. 3. Grilli R, Magrini N, Penna A, Mura G and Liberati A: Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 2000; 355: 103. 4. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J and Deshpande AM: Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med 2003; 139: 493. 5. National Guideline Clearinghouse. Agency for Healthcare Research and Quality (AHRQ). Available at www.guideline.gov. 6. Krupski TL, Dahm P, Fesperman SF and Schardt CM: How to perform a literature search. J Urol 2008; 179: 1264. 7. Thompson I, Thrasher JB, Aus G, Burnett AL, Canby-Hagino ED, Cookson MS et al: Guideline for the management of clinically localized prostate cancer: 2007 update. J Urol 2007; 177: 2106.
lines: the AGREE project. Qual Saf Health Care 2003; 12: 18. 13. Oxman AD, Schunemann HJ and Fretheim A: Improving the use of research evidence in guideline development. 14. Reporting guidelines. Health Res Policy Syst 2006; 4: 26. 14. Hayward RS, Wilson MC, Tunis SR, Bass EB and Guyatt G: Users’ guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? The EvidenceBased Medicine Working Group. JAMA 1995; 274: 570. 15. Guyatt GH, Oxman AD, Kunz R, Vist GE, FalckYtter Y and Schunemann HJ: What is “quality of evidence” and why is it important to clinicians? BMJ 2008; 336: 995. 16. Drummond MF, Richardson WS, O’Brien BJ, Levine M and Heyland D: Users’ guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA 1997; 277: 1552. 17. O’Brien BJ, Heyland D, Richardson WS, Levine M and Drummond MF: Users’ guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. B. What are the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. JAMA 1997; 277: 1802.
8. Thompson I, Thrasher JB, Aus G, Burnett AL, Canby-Hagino ED, Cookson MS et al: Guideline for the Management of Clinically Localized Prostate Cancer: 2007 Update. American Urological Association 2007; p 82.
18. Guyatt GH, Oxman AD, Kunz R, Jaeschke R, Helfand M, Liberati A et al: Incorporating considerations of resources use into grading recommendations. BMJ 2008; 336: 1170.
9. Guyatt GH and Rennie D: User’s Guide to the Medical Literature, 4th ed. Chicago: AMA Press 2002; p 706.
19. Oxman AD, Cook DJ and Guyatt GH: Users’ guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. JAMA 1994; 272: 1367.
10. Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ and Cook RJ: Users’ guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group. JAMA 1995; 274: 1800. 11. Vlayen J, Aertgeerts B, Hannes K, Sermeus W and Ramaekers D: A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care 2005; 17: 235. 12. Collaboration AGREE : Development and validation of an international appraisal instrument for assessing the quality of clinical practice guide-
20. Montori VM, Devereaux PJ, Adhikari NK, Burns KE, Eggert CH, Briel M et al: Randomized trials stopped early for benefit: a systematic review. JAMA 2005; 294: 2203. 21. Moher D, Schulz KF and Altman D: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001; 285: 1987. 22. Montori VM and Guyatt GH: Intention-to-treat principle. CMAJ 2001; 165: 1339. 23. Krahn M and Naglie G: The next step in guideline development: incorporating patient preferences. JAMA 2008; 300: 436.
24. Shekelle PG, Kravitz RL, Beart J, Marger M, Wang M and Lee M: Are nonspecific practice guidelines potentially harmful? A randomized comparison of the effect of nonspecific versus specific guidelines on physician decision making. Health Serv Res 2000; 34: 1429. 25. Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A et al: Going from evidence to recommendations. BMJ 2008; 336: 1049. 26. Schunemann HJ, Fretheim A and Oxman AD: Improving the use of research evidence in guideline development: 9. Grading evidence and recommendations. Health Res Policy Syst 2006; 4: 21. 27. Dahm P, Yeung LL, Chang SS and Cookson MS: A critical review of clinical practice guidelines for the management of clinically localized prostate cancer. J Urol 2008; 180: 451. 28. Eddy D: A Manual For Assessing Health Practices And Designing Practice Policies: The Explicit Approach. Philadelphia: American College of Physicians 1992; p 126. 29. D’Amico AV, Whittington R, Malkowicz SB, Fondurulia J, Chen MH, Kaplan I et al: Pretreatment nomogram for prostate-specific antigen recurrence after radical prostatectomy or externalbeam radiation therapy for clinically localized prostate cancer. J Clin Oncol 1999; 17: 168. 30. Sur RL, Scales CD Jr, Preminger GM and Dahm P: Evidence-based medicine: a survey of American Urological Association members. J Urol 2006; 176: 1127. 31. Guyatt G, McAlister F, Haynes RB, Sinclair J, Devereaux PJ and Lacchetti C: Incorporating patient values. In: Users’ Guide to the Medical Literature. Edited by G Guyatt. Chicago: AMA Press 2002. 32. Guyatt G, Montori V, Devereaux PJ, Schunemann H and Bhandari M: Patients at the center: in our practice, and in our use of language. ACP J Club 2004; 140: A11. 33. Gonzalez CM, Penson D, Kosiak B, Dupree J and Clemens JQ: Pay for performance: rationale and potential implications for urology. J Urol 2007; 178: 402. 34. Miller DC, Wei JT, Montie JE and Hollenbeck BK: Quality of care and performance-based reimbursement: the contemporary landscape and implications for urologists. Urology 2006; 67: 1117.
How to Use a Clinical Practice Guideline Philipp Dahm,*,† Lawrence L. Yeung,† Michele Gallucci, Giuseppe Simone and Holger J. Schünemann From the Department of Urology, College of Medicine, University of Florida, Gainesville, Florida (PD, LLY), Department of Urology (MG) and Epidemiology (HJS), Italian National Cancer Institute Regina Elena, Rome, and Department of Urology, Magna Graecia University of Catanzaro, Catanzaro (GS), Italy
Abbreviations and Acronyms AUA ⫽ American Urological Association Submitted for publication August 5, 2008. Nothing to disclose. * Correspondence: Department of Urology, University of Florida College of Medicine, Health Science Center, Box 100247, Room M2-204, Gainesville, Florida 32610-0247 (telephone: 352273-6815; FAX: 352-273-7515; e-mail: p.dahm@ urology.ufl.edu). † Equal study contribution.
Editor’s Note: This article is the second of 5 published in this issue for which category 1 CME credits can be earned. Instructions for obtaining credits are given with the questions on pages 928 and 929.
Purpose: Leading organizations increasingly recognize clinical practice guidelines as an important approach in promoting an evidence-based clinical practice of urology. In light of their considerable clinical, economic and medicolegal impact, guideline users should be confident that the guidelines were rigorously developed and address relevant patient questions. In this article we outline a practical approach for critically appraising a clinical practice guideline. Materials and Methods: We outline a 3-step approach to the assessment of a clinical practice guideline that answers the questions of whether the recommendations are valid, what the recommendations are and whether they will help in the treatment of an individual patient. Results: To determine the adequacy of a clinical practice guideline, urologists should carefully review the rigor of the development process and its content. Important questions that relate to the validity of a guideline are whether, for specific questions, all important management options and outcomes were considered, and whether there was an explicit and sensible process to identify, select and combine all relevant evidence. Clinical practice guidelines should formally grade the quality of the available evidence for a given clinical question and outline a formal process of how the recommendations were derived. Value judgments made in the guideline development process about the relative importance of the potential benefits and harms of a given health care intervention should be made transparent to the reader. The recommendations made should be practical and should address important clinical issues. Furthermore, their strength should be graded to reflect the underlying uncertainty about the evidence and the values applied in the guideline development process. Conclusions: The systematic approach presented in this article will allow urologists to critically appraise clinical practice guidelines. Determining the validity of the recommendations, understanding the recommendations and assessing their applicability to patients are 3 fundamental steps toward an evidence-based approach to using clinical practice guidelines. Key Words: practice guidelines as topic, evidence-based medicine
472
www.jurology.com
UROLOGISTS are continuously faced with the need to make or facilitate decision making for individual patients. Making treatment recommendations implicitly or explicitly involves formulating a clinical question, identifying management
options and outcomes, collecting and summarizing evidence, and applying value judgments and preferences to arrive at an optimal treatment plan. In an attempt to facilitate clinical decision making and promote the delivery of
0022-5347/09/1812-0472/0 THE JOURNAL OF UROLOGY® Copyright © 2009 by AMERICAN UROLOGICAL ASSOCIATION
Vol. 181, 472-479, February 2009 Printed in U.S.A. DOI:10.1016/j.juro.2008.10.041
HOW TO USE CLINICAL PRACTICE GUIDELINES
consistent, high quality care to all patients, clinical practice guidelines are being developed by various local, regional and international organizations. Clinical practice guidelines seek to provide explicit recommendations for the treatment of typical index patients based on the best available research evidence, while at the same time assessing the balance between desirable and undesirable effects, the patient’s presumed values and preferences, and the potential economic impact of different treatment options. As such, the Institute of Medicine defines them as “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.”1 Unfortunately the methodological quality of clinical practice guidelines developed by different organizations varies considerably, thereby impacting the confidence that we can place in their recommendations.1– 4 Therefore, it is important that urologists be familiar with the defining characteristics of clinical guidelines that deserve the label evidence-based, and that they have a conceptual framework to determine whether a guideline document is likely to be valid, to interpret the recommendations appropriately and to determine whether the recommendations are applicable to a particular patient. Thus, we introduce a systematic approach to critically evaluate a clinical practice guideline document before applying its recommendations to the care of an individual patient.
CLINICAL SCENARIO You are a community practice urologist and are previewing your list of clinic patients for the next day which includes the brother of a close friend. Your friend asked for a second opinion for this 61-year-old patient with newly diagnosed prostate cancer. Outside records have been faxed to your office. The patient has clinical T1c disease with Gleason score 3 ⫹ 3 ⫽ 6 adenocarcinoma of the prostate in 4 of 14 cores and a prostate specific antigen of 4.5 ng/ml. The urological history is negative for any significant urinary complaints but mentions poor erectile function. The medical history is notable for mild hypertension, hyperlipidemia and adult onset diabetes which is well controlled with an oral agent. Having recently attended a urological meeting where much emphasis was placed on the importance of evidence-based decision making consistent with clinical practice guidelines, you feel compelled to update your knowledge on the management of clinically localized prostate cancer. Therefore, you decide to retrieve the most recent clinical practice guidelines on prostate cancer to guide the treatment of this patient.
473
SEARCH OF THE MEDICAL LITERATURE From your reading of the “Users’ Guide to the Urological Literature” by Krupski et al on how to search the medical literature effectively, you recall the National Guideline Clearing House as a dedicated resource for evidence-based clinical practice guidelines.5,6 Using prostate cancer as single search term you find 65 guideline documents (accessed June 1, 2008). Considering recent advances in the understanding of the natural history of prostate cancer as well as new treatment approaches including active surveillance, you decide to narrow the search by applying filters for treatment (vs diagnosis or prognosis) and the last 5 years only (2004 to 2008), which yields 27 guidelines. Among these only 1 document with the title “Guidelines for the Management of Clinically Localized Prostate Cancer: 2007 Update” strikes you as truly applicable to your patient scenario.7,8 Delighted to have found a recent guideline by a leading urological association you decide to print a copy of this document and review it before your patient’s clinic visit.
ARE THE RECOMMENDATIONS VALID? Before using evidence provided by a clinical research study, urologists should ask whether the results of the study are likely to be valid.9 The same principle applies to clinical practice guidelines.10 Guideline users should be confident that these recommendations are the product of a rigorous evidence driven development process, that the recommendations address clinically relevant issues and are applicable to the patient for which guidance is being sought. To assess the methodological quality of clinical practice guidelines several validated instruments have been developed.11 Among them the Appraisal of Guidelines for Research and Evaluation (AGREE) collaboration instrument is one of the best known and most widely used.12 It distinguishes 6 domains of methodological quality and transparent reporting. 1) “Scope and purpose” deals with the overall aim of the guideline, the specific clinical questions and the target patient population. 2) “Stakeholder involvement” focuses on the extent to which the guideline represents the views of its intended users. 3) “Rigor of development” relates to the process used to gather and synthesize the evidence and the methods to formulate the recommendations, and to update them. 4) “Clarity and presentation” deals with the language and format of the guideline. 5) “Applicability” pertains to the likely organizational, behavioral and cost implications of applying the guideline. 6) “Editorial independence” addresses the independence of the recommendations and acknowledgment of possible conflict of interest from the guideline development group. A similar endeavor to establish a shared framework for guideline development, reporting and assessment
474
HOW TO USE CLINICAL PRACTICE GUIDELINES
has been put forward by COGS (the Conference on Guideline Standardization). This group, with representatives from 22 professional organizations, used a 2-stage modified Delphi process to develop standards for reporting clinical practice guidelines which resulted in an 18-item COGS checklist as a consensus standard for reporting.4,13 Also, the guideline content presented in the National Guideline Clearinghouse represents a standard for reporting imposed on organizations that want their guidelines included in that database. It is composed of 52 items including the topic scope, methodology, evidence supporting the recommendations, benefits/harms of implementing the recommendations, contraindications and implementation of the guideline among others. However, for the purpose of the clinical reader a more rapid assessment of study validity relies on a few key questions (Appendix 1). Were All Important Options and Outcomes Considered? Guidelines pertain to clinical decision making, and decisions involve choices and consequences.14 To understand why a particular practice is recommended you should consider whether the guideline developers included all reasonable practice options and important consequences. For example, when it comes to the treatment of patients with clinically localized prostate cancer you expect guideline developers to have considered radical surgery, radiation therapy, active surveillance and potentially cryotherapy as therapeutic options. Important patient relevant outcomes include long-term survival, quality of life, urinary and sexual function as well as information about the morbidity associated with different options. Ideally guideline developers should also present a hierarchy of the important patient outcomes that they considered, and distinguish between critical and noncritical outcomes for clinical decision making.15 This distinction should be made at an early stage of the guideline development process. As a cost conscious provider you should also look for economic considerations in the guideline document.16,17 When economic outcomes are taken into account it is important to realize which perspective was chosen, such as that of the patient, insurer or health care systems overall, and whether broader issues such as the consequences of time lost from work were included which can have a major impact on the final recommendations.14,18 Was an Explicit and Sensible Process Used to Identify, Select and Combine Evidence? Once patient relevant outcomes and interventions have been defined, the next task in decision making is to estimate the likelihood that a given outcome will occur. To do so guideline developers should define a series of specific questions.15 For example, one such focused clinical question would be, “In patients with clinically localized prostate cancer, how does treat-
ment with radical prostatectomy compare to active surveillance with regard to long-term survival?”. The structure of this question follows the same Population, Intervention, Comparison and Outcome (PICO) format introduced by Krupski et al.6 Guideline developers must then bring together all relevant evidence and combine it in an appropriate manner that avoids the introduction of bias. The criteria by which to judge whether guideline developers have done an adequate job in accumulating and synthesizing the evidence are similar to those that apply to systematic reviews.19 Guideline developers should specify a focused question, define appropriate evidence using explicit inclusion and exclusion criteria, conduct a comprehensive search and examine the validity of the results in a reproducible fashion. Guidelines should report what type of evidence was considered, report how it was selected, abstracted and combined, and make key data available for review. We will review important methodological criteria by which to assess the rigor of the guideline development process. Is There a Description of Individuals Involved in the Development Process? For clinical practice guidelines to provide meaningful recommendations it is important that explicit and implicit judgments made by a panel reflect the views of its intended users. The guideline panel is responsible for interpreting the published evidence with regard to their own knowledge and experience in their respective fields and, therefore, should have broad representation. Multispecialty panels involving members with a variety of backgrounds such as patients, primary care providers, specialists, nonphysician providers (eg nurses, physical therapists, patient advocates), research experts and health care policy makers are more likely to produce balanced guidelines than a panel composed of only urologists, for example. A panel with a varied background will provide a diversity of experience and philosophy on health care, thereby limiting potential bias in the development process. Were Conflicts of Interest of the Guideline Panel Members Considered? In light of the considerable implications that guideline recommendations for or against certain treatment modalities may have, the guideline panel should provide assurance to potential users that the development process was not influenced by special interest groups. Although it may be beneficial for the guideline panel to include clinical investigators who have served as consultants or principal investigators of industry sponsored trials, any potential conflicts of interest need to explicitly identified. In addition, there needs to be mechanisms in place to manage any conflicts of interest should they arise.
HOW TO USE CLINICAL PRACTICE GUIDELINES
Are the Sources of Evidence Used in the Guideline Development Identified? Clinical practice guidelines should be based on a comprehensive and systematic review of the entire body of evidence, published and unpublished, that is applicable to the clinical question of interest. Details of the strategy used to obtain the evidence should be explicitly stated including the search terms used, the sources used and the time frame of the literature that was incorporated into the clinical practice guideline. Criteria for including or excluding evidence identified by the search should also be provided. What Were the Methods Used to Interpret and Assess the Strength of Evidence? One of the guiding principles of evidence-based medicine is the concept of a hierarchy of evidence. It recognizes that certain study designs are more suitable than others to provide an accurate representation of the truth, ie be less biased. For questions regarding therapeutic effectiveness, randomized controlled trials provide the highest level of evidence. Ideally, these randomized controlled trials show similar findings (ie low heterogeneity) and are summarized in systematic reviews that contain meta-analyses. However, individual randomized controlled trials need to address certain methodological safeguards against bias to be considered valid. Major limitations that may bias the estimates of the treatment effect include failure to implement allocation concealment, lack of blinding especially if the outcomes are subjective and their assessment highly susceptible to bias, and large losses to followup. Additional issues include the failure to adhere to intent to treat analysis and the issue of stopping trials early for apparent benefit.20 –22 Therefore, guideline developers need to critically analyze the methodological quality of the studies on which they base their recommendations rather than relying on the study design alone, and downgrade the evidence that is derived from studies with serious methodological flaws. Were the Methods Used to Formulate Recommendations Described? Once the quality of evidence has been established, guideline developers need to integrate and synthesize the available evidence to generate recommendations that provide specific guidance for clinicians, patients and policy makers. The strength of recommendations should be graded to reflect the degree of certainty that the desirable effects of adherence to a recommendation outweigh the undesirable effects. Key considerations that determine the strength of a recommendation include the quality of the evidence, the balance between desirable and undesirable consequences of alternative management strategies, the degree of uncertainty or variability of patient values and preferences for which the guideline document is intended and finally, the
475
associated costs.15,23 The guideline process that is used to integrate these factors may vary by the methodological framework that is used, but needs to be transparent and reproducible. Specifically the judgments that panelists form during the guideline development process need to be explicit to the user. These judgments relate to the relative importance of certain outcomes compared to others and assumptions that are being made about patient preferences. Has the Guideline Been Subjected to Peer Review and Testing? Similar to any peer reviewed publication, clinical practice guidelines should be externally reviewed before their public dissemination. The purpose of this external review is to provide other individuals with vested interests and expertise in this area to scrutinize the rigor of the guideline development process, as well as the appropriateness and applicability of the recommendations being made. The external review should be considered an integral part of the guideline development process, follow clearly defined guidelines and be clearly documented. In addition, it is highly recommended to test the application of a clinical practice guideline in a sample of patients that is similar to the population to which the guidelines will ultimately apply. Pilot testing provides valuable insights into the practical applicability of a guideline document and helps to understand its potential downstream effects. Are Methods for Updating the Guideline Document Described? Clinical practice guidelines should be based on the current best available evidence. Since new research evidence is published on a daily basis, guidelines invariably become outdated unless there are mechanisms in place to keep them updated. The frequency of such updates would be dictated by how rapidly new research findings become available in a certain field. Therefore, guidelines should have a mechanism in place to ensure that the recommendations are routinely reviewed and do not become outdated. Validity Assessment of the Clinical Practice Guideline on Prostate Cancer The AUA 2007 guideline document on clinically localized prostate cancer appears to address all patient relevant outcomes except resource use.7,8 It describes the individuals who were involved in the guidelines development as a multidisciplinary team. The evidence source is clearly identified although limited to the English literature that was indexed by the National Library of Medicine. The process of data abstraction is outlined and several quality checks were included to assure accuracy. The data were then compiled and presented to the guideline panel for discussion. No level of evidence grading system was applied to studies considered by the guideline panel, and no
476
HOW TO USE CLINICAL PRACTICE GUIDELINES
formal process as to how the recommendations were formulated has been reported. Although not pilot tested, the AUA guidelines were peer reviewed. The guidelines document addresses how the guidelines will be disseminated. A mechanism for updating the guidelines appears in place. In summary, shortcomings of the AUA guidelines are the failure to consider economic implications, clearly define the quality of underlying evidence and describe a formal process by which the recommendations were formulated. Meanwhile, many other important validity criteria appear to be met. Therefore, we would conclude that recommendations are likely to be valid with the previously mentioned reservations and, thus, will review them using a similar systematic approach.
WHAT ARE THE RECOMMENDATIONS? Once you have determined that the guideline recommendations are likely to be valid, you should review the actual recommendations that are being made. Are Practical, Clinically Important Recommendations Made? Clinical practice guidelines are developed to assist clinicians, patients and health policy makers in clinical decision making. Therefore, guideline recommendations need to provide practical, unambiguous advice for clearly defined problems that arise in the treatment of patients.4 Shekelle et al evaluated the effect of different levels of specificity of recommendations on clinicians’ test ordering behavior using clinical vignettes.24 They found that clinicians receiving the nonspecific recommendations ordered fewer indicated tests for appropriate indications than physicians who received specific recommendations. The authors concluded that the clarity and clinical applicability of a guideline are important attributes that contribute to the effects of practice guidelines. How Strong are the Recommendations? The strength, grade, confidence or force of recommendations made by guideline developers reflects the extent to which we can be confident that the desirable effects of an intervention outweigh the undesirable effects.25 In addition, the strength of a recommendation reflects considerations about the quality of the evidence, the necessary resources, and the extent by which the values and preferences of patients regarding the importance of certain outcomes vary. Therefore, the grading of recommendations is an important way for guideline developers to convey this level of confidence to the user. While most guideline development groups now grade the quality of evidence and the strength of recommendations, a variety of different grading systems are being used, thereby making it difficult to compare recommendations by different or-
ganizations and possibly impeding guideline implementation.26,27 Summary of Recommendations Made by the AUA Practice Guidelines The treatment options in the 2007 AUA guidelines for localized prostate cancer address several practically important management decisions. At the same time many important issues in clinically localized prostate cancer that clinicians and patients face on a day-to-day basis remain unaddressed. For example, the guideline document fails to provide any guidance on issues relevant to urological surgeons such as when to choose one surgical approach over another or which patients should undergo lymph node dissection for clinically localized prostate cancer. The AUA guidelines graded the strength of its recommendations using a 3-tiered system of flexibility based on the strength of evidence and the panel assessment of patient needs and preferences as standards, recommendations and options which are intended to signal to users the level of confidence that should be placed into these recommendations (Appendix 2).28 Recommendations were labeled standard if the panel concluded that it should be followed by virtually all health care providers for virtually all patients. However, few of the recommendations were considered standards which are intended to provide strong guidance for clinical decision making. The guideline panel formulated a recommendation if the members thought it was appropriate not for all, but for a significant majority of patients. Recommendations were graded as option if the panel determined that the health outcomes were not sufficiently wellknown and no clear patient preferences were apparent. One major underlying issue for the failure to address other important clinical questions and the paucity of strong recommendations is clearly stated in the guideline document, and relates to a lack of high quality evidence for many important questions in prostate cancer.27 However, there are also considerable issues with the system used for grading the strength of the recommendation itself. A recommendation labeled as an option provides little guidance to the clinician at all and, therefore, could be eliminated. In addition, the distinction between standard (which should be applied to virtually all patients) and recommendation (applicable to a significant majority) is ambiguous. Finally, based on the definitions provided there is not a single situation in which clinicians and patients could not deviate from a standard or recommendation. Therefore, it appears that the ambiguity of the recommendations would limit their practical applicability in the clinical care of patients.
HOW TO USE CLINICAL PRACTICE GUIDELINES
WILL THE RECOMMENDATIONS HELP YOU IN CARING FOR YOUR PATIENTS? Having assessed the validity of the guideline development process and reviewed the actual recommendations being made you should next ask whether the clinical guideline will actually help you in treating your patients by considering 3 questions. Is the Primary Objective of the Guideline Consistent With Your Objectives? Clinical practice guidelines are being developed for different purposes and different target audiences. Most commonly we think of guidelines as instruments to assist physicians and patients with clinical decision making. However, guidelines are also being developed to evaluate physician practices for quality assurance purposes or to set limits with regard to physician choices for reimbursement. Prostate cancer guidelines may also target other types of health care providers such as radiation oncologists, medical oncologists and nurses. Therefore, you should ensure that the purpose of the guideline is consistent with the use you intend for it. Are the Recommendations Applicable to Your Patients? To be really useful guidelines should describe interventions or procedures well enough for their exact duplication. You must also determine whether your patient characteristics fit the intended target audience of the guideline. If your patient has a different prevalence of disease or risk factors, for instance, the guideline may not apply. Ideally in the face of important differences in patient or practice characteristics, guideline flexibility allows users to individualize recommendations or justify departure from the recommendations altogether. For example, the risk stratification of clinically localized prostate cancer into low, intermediate and high risk categories by the AUA guidelines is an example of differing recommendations for different prognostic groups. Finally, you should look for information about the values that were assigned explicitly or implicitly to alternative outcomes. If the assigned values differ sufficiently from patient preference, this discrepancy may impact the decision to implement a recommendation. Applicability of AUA Guidelines to the Patient in the Clinical Scenario A review of the AUA guidelines document confirms that its primary purpose is to guide provider and patient decision making in patients with clinically localized prostate cancer. Developed by the AUA, urologists are the main target audience and the guideline objectives appear to be consistent with your own. In addition, your patient appears to be
477
similar to those for whom the guideline was developed, and likely shares similar values and preferences. Therefore, you conclude that the AUA guideline document is likely applicable to your patient.
RESOLUTION OF CLINICAL SCENARIO After a thorough review of the 2007 AUA guidelines on clinically localized prostate cancer you feel well prepared to counsel this patient. A thorough review of patient history of present illness, medical history and review of systems confirms the information you were provided. Based on the available clinical information, you determine that the patient has a statistical life expectancy of well over 10 years and falls into the low risk category based on the D’Amico classification system and, therefore, is a good candidate for local treatment with curative intent.29 To decide whether you can and should apply the AUA guidelines to the care of this patient you once again consider the 3 central questions recommended by the Users’ Guide to the Urological Literature. With regard to the validity of the guidelines you determine that the guideline document is more likely to be valid than not. Key shortcomings were found in the lack of formal grading of the quality of evidence, and a transparent and well-defined process of how guideline developers formulated the recommendations. With regard to what the recommendations are, you found the system used to grade the strength of recommendations to be ambiguous and of limited practical value. In addition, several clinically important questions remained unaddressed, likely because the evidence to address them was lacking. Finally, with regard to the question of whether the guideline document will actually help you in caring for your patient, you found its objectives to be consistent with your own and the guideline document applicable to your patient. In summary, your critical review of the guideline has identified a series of issues which hinder its use in clinical decision making and limit its practical value. That said, the guideline document provides a comprehensive overview of available treatment options as well as their associated benefits and potential side effects. Also, many of the recommendations defined as a standard appear sensible and noncontroversial. Specifically you decide to follow the guideline recommendation to provide your patient with a comprehensive overview of available treatment options including active surveillance, radiotherapy and radical surgery. In addition, you dedicate a considerable amount of time exploring patient opinions and preferences with regard to the potential urinary, sexual and bowel adverse events as the result of treatment. You learn through this conver-
478
HOW TO USE CLINICAL PRACTICE GUIDELINES
sation that he is anxious about delaying curative treatment due to concerns about disease progression and, therefore, reluctant about active surveillance. He places little importance on erectile function due to his marital situation and not having been sexually active for many years. Finally he expresses serious concerns about the risk of bowel related side effects whereas the risk of some degree of stress urinary incontinence after surgery is less threatening to him. Nevertheless, you suggest he see a radiation oncologist for consideration of primary radiation treatment, to which he agrees. You also provide him with a copy of the patient guide for the management of localized prostate cancer which is published by the AUA as a supporting document for the full guidelines document. Several weeks later he returns to your office for further counseling at which time he indicates his decision to undergo radical prostatectomy. Surgery and postoperative course are uneventful, and the patient experiences a timely recovery of urinary function and no short-term complications. The patient is most grateful for the treatment he received, and specifically thanks you for the time and effort you spent counseling him to guide his decision making.
CONCLUSIONS Clinical practice guidelines are becoming increasingly relevant evidence-based resources to guide clinical decision making for urologists.30 This article provides a basic framework for the critical appraisal of clinical practice guidelines using the AUA guidelines for clinically localized prostate cancer as an example. Guideline users should ask whether the recommendations are valid, what the recommendations are and whether they can be applied to the care of individual patients. These 3 questions are independently important and should be answered affirmatively before the guidelines are applied to actual patient care. Guidelines that meet these criteria can be an excellent guide for individual clinical decision making when integrated with the individual values and preferences of the patient.23,31,32 Evidence-based clinical practice guidelines also provide the basis of quality of care measures which are currently under development and likely to impact every urology practice in the future.33,34 Therefore, guideline methodology is of paramount importance and the subject of considerable scrutiny. Recognizing the importance of a transparent and evidence-based guideline development process, and
following careful assessment of various methodological approaches, the AUA has recently announced a major overhaul of its guidelines development process. The revised process is expected to address many of the shortcomings highlighted in this article which include explicit grading of the quality of evidence and clearer definitions of the strengths of recommendations. Efforts to move towards a unified approach to guideline development by all urological associations appear increasingly important. A joint guideline development process would provide tremendous opportunities for international consensus building, cost reduction and ultimately improved quality of care for all urological patients.
APPENDIX 1 Users’ Guide to a Clinical Practice Guideline Are the recommendations valid? Were all important options and outcomes considered? Was an explicit and sensible process used to identify, select and combine evidence? Is there a description of individuals involved in the development process? Were the conflicts of interest of the guideline panel members accounted for? Are the sources of evidence used in the guideline development identified? What were the methods used to interpret and assess the strength of evidence? Were the methods used to formulate recommendations described? Has the guideline been subjected to peer review and testing? Are methods for updating the guideline document described? What are the recommendations? Are practical, clinically important recommendations made? How strong are the recommendations? Will the recommendations help you in caring for your patients? Is the primary objective of the guideline consistent with your objectives? Are the recommendations applicable to your patients?
APPENDIX 2 Three-Tiered System Used by the 2007 AUA Guidelines on Clinically Localized Prostate Cancer to Grade the Strength of Recommendations Standard
Recommendation
Option
A guideline statement is a standard if: (1) the health outcomes of the alternative interventions are sufficiently well-known to permit meaningful decisions, and (2) there is virtual unanimity about which intervention is preferred. A guideline statement is a recommendation if: (1) the health outcomes of the alternative interventions are sufficiently well-known to permit meaningful decisions, and (2) an appreciable but not unanimous majority agrees on which intervention is preferred. A guideline statement is an option if: (1) the health outcomes of the interventions are not sufficiently well-known to permit meaningful decisions, or (2) preferences are unknown or equivocal.
HOW TO USE CLINICAL PRACTICE GUIDELINES
479
REFERENCES 1. Shaneyfelt TM, Mayo-Smith MF and Rothwangl J: Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 1999; 281: 1900. 2. Guyatt G, Vist G, Falck-Ytter Y, Kunz R, Magrini N and Schunemann H: An emerging consensus on grading recommendations? ACP J Club 2006; 144: A8. 3. Grilli R, Magrini N, Penna A, Mura G and Liberati A: Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 2000; 355: 103. 4. Shiffman RN, Shekelle P, Overhage JM, Slutsky J, Grimshaw J and Deshpande AM: Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med 2003; 139: 493. 5. National Guideline Clearinghouse. Agency for Healthcare Research and Quality (AHRQ). Available at www.guideline.gov. 6. Krupski TL, Dahm P, Fesperman SF and Schardt CM: How to perform a literature search. J Urol 2008; 179: 1264. 7. Thompson I, Thrasher JB, Aus G, Burnett AL, Canby-Hagino ED, Cookson MS et al: Guideline for the management of clinically localized prostate cancer: 2007 update. J Urol 2007; 177: 2106.
lines: the AGREE project. Qual Saf Health Care 2003; 12: 18. 13. Oxman AD, Schunemann HJ and Fretheim A: Improving the use of research evidence in guideline development. 14. Reporting guidelines. Health Res Policy Syst 2006; 4: 26. 14. Hayward RS, Wilson MC, Tunis SR, Bass EB and Guyatt G: Users’ guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? The EvidenceBased Medicine Working Group. JAMA 1995; 274: 570. 15. Guyatt GH, Oxman AD, Kunz R, Vist GE, FalckYtter Y and Schunemann HJ: What is “quality of evidence” and why is it important to clinicians? BMJ 2008; 336: 995. 16. Drummond MF, Richardson WS, O’Brien BJ, Levine M and Heyland D: Users’ guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA 1997; 277: 1552. 17. O’Brien BJ, Heyland D, Richardson WS, Levine M and Drummond MF: Users’ guides to the medical literature. XIII. How to use an article on economic analysis of clinical practice. B. What are the results and will they help me in caring for my patients? Evidence-Based Medicine Working Group. JAMA 1997; 277: 1802.
8. Thompson I, Thrasher JB, Aus G, Burnett AL, Canby-Hagino ED, Cookson MS et al: Guideline for the Management of Clinically Localized Prostate Cancer: 2007 Update. American Urological Association 2007; p 82.
18. Guyatt GH, Oxman AD, Kunz R, Jaeschke R, Helfand M, Liberati A et al: Incorporating considerations of resources use into grading recommendations. BMJ 2008; 336: 1170.
9. Guyatt GH and Rennie D: User’s Guide to the Medical Literature, 4th ed. Chicago: AMA Press 2002; p 706.
19. Oxman AD, Cook DJ and Guyatt GH: Users’ guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. JAMA 1994; 272: 1367.
10. Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ and Cook RJ: Users’ guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group. JAMA 1995; 274: 1800. 11. Vlayen J, Aertgeerts B, Hannes K, Sermeus W and Ramaekers D: A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care 2005; 17: 235. 12. Collaboration AGREE : Development and validation of an international appraisal instrument for assessing the quality of clinical practice guide-
20. Montori VM, Devereaux PJ, Adhikari NK, Burns KE, Eggert CH, Briel M et al: Randomized trials stopped early for benefit: a systematic review. JAMA 2005; 294: 2203. 21. Moher D, Schulz KF and Altman D: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001; 285: 1987. 22. Montori VM and Guyatt GH: Intention-to-treat principle. CMAJ 2001; 165: 1339. 23. Krahn M and Naglie G: The next step in guideline development: incorporating patient preferences. JAMA 2008; 300: 436.
24. Shekelle PG, Kravitz RL, Beart J, Marger M, Wang M and Lee M: Are nonspecific practice guidelines potentially harmful? A randomized comparison of the effect of nonspecific versus specific guidelines on physician decision making. Health Serv Res 2000; 34: 1429. 25. Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A et al: Going from evidence to recommendations. BMJ 2008; 336: 1049. 26. Schunemann HJ, Fretheim A and Oxman AD: Improving the use of research evidence in guideline development: 9. Grading evidence and recommendations. Health Res Policy Syst 2006; 4: 21. 27. Dahm P, Yeung LL, Chang SS and Cookson MS: A critical review of clinical practice guidelines for the management of clinically localized prostate cancer. J Urol 2008; 180: 451. 28. Eddy D: A Manual For Assessing Health Practices And Designing Practice Policies: The Explicit Approach. Philadelphia: American College of Physicians 1992; p 126. 29. D’Amico AV, Whittington R, Malkowicz SB, Fondurulia J, Chen MH, Kaplan I et al: Pretreatment nomogram for prostate-specific antigen recurrence after radical prostatectomy or externalbeam radiation therapy for clinically localized prostate cancer. J Clin Oncol 1999; 17: 168. 30. Sur RL, Scales CD Jr, Preminger GM and Dahm P: Evidence-based medicine: a survey of American Urological Association members. J Urol 2006; 176: 1127. 31. Guyatt G, McAlister F, Haynes RB, Sinclair J, Devereaux PJ and Lacchetti C: Incorporating patient values. In: Users’ Guide to the Medical Literature. Edited by G Guyatt. Chicago: AMA Press 2002. 32. Guyatt G, Montori V, Devereaux PJ, Schunemann H and Bhandari M: Patients at the center: in our practice, and in our use of language. ACP J Club 2004; 140: A11. 33. Gonzalez CM, Penson D, Kosiak B, Dupree J and Clemens JQ: Pay for performance: rationale and potential implications for urology. J Urol 2007; 178: 402. 34. Miller DC, Wei JT, Montie JE and Hollenbeck BK: Quality of care and performance-based reimbursement: the contemporary landscape and implications for urologists. Urology 2006; 67: 1117.