- Email: [email protected]
Implementation of Nephrology Subspecialty Curricular Milestones
Perspective Implementation of Nephrology Subspecialty Curricular Milestones Christina M. Yuan, MD, Lisa K. Prince, MD, James D. Oliver III, MD, PhD, Kevin C. Abbott, MD, MPH, and Robert Nee, MD Beginning in the 2014-2015 training year, the US Accreditation Council for Graduate Medical Education (ACGME) required that nephrology Clinical Competency Committees assess fellows’ progress toward 23 subcompetency “context nonspecific” internal medicine subspecialty milestones. Fellows’ advancement toward the “ready for unsupervised practice” target milestone now is tracked in each of the 6 competencies: Patient Care, Medical Knowledge, Professionalism, Interpersonal Communication Skills, Practice-Based Learning and Improvement, and Systems-Based Practice. Nephrology program directors and subspecialty societies must define nephrology-specific “curricular milestones,” mapped to the nonspecific ACGME milestones. Although the ACGME goal is to produce data that can discriminate between successful and underperforming training programs, the approach is at risk to produce biased, inaccurate, and unhelpful information. We map the ACGME internal medicine subspecialty milestones to our previously published nephrologyspecific milestone schema and describe entrustable professional activities and other objective assessment tools that inform milestone decisions. Mapping our schema onto the ACGME subspecialty milestone reporting form allows comparison with the ACGME subspecialty milestones and the curricular milestones developed by the American Society of Nephrology Program Directors. Clinical Competency Committees may easily adapt and directly translate milestone decisions reached using our schema onto the ACGME internal medicine subspecialty competency milestone-reporting format. Am J Kidney Dis. 66(1):15-22. Published by Elsevier Inc. on behalf of the National Kidney Foundation, Inc. This is a US Government Work. There are no restrictions on its use. INDEX WORDS: Nephrology; curricular milestones; fellowship; medical trainee; training programs; graduate medical education; competencies; subspecialty milestones; ACGME.
I
n 2012, the Accreditation Council for Graduate Medical Education (ACGME) introduced the Next Accreditation System.1 One objective of this system is to base training program accreditation on trainee outcomes, using milestone attainment in the 6 basic competencies: Patient Care (PC), Medical Knowledge (MK), Professionalism (PROF), Interpersonal Communication Skills (ICS), Practice-Based Learning and Improvement (PBLI), and SystemsBased Practice (SBP). Milestones are competencybased progressive achievements, demonstrated by trainees during the period of their education and culminating in graduation and the commencement of unsupervised practice. Aggregate performance of program trainees in achieving milestones will be one outcome data element used annually by the Next Accreditation System to assess programmatic success/ compliance, ultimately producing national competency outcomes within specialties and subspecialties. In March 2014, the ACGME published “context nonspecific” internal medicine subspecialty milestones in 23 subcompetencies (Box 1).2 Clinical Competency Committees (CCCs) will submit twice-yearly trainee evaluations using the subcompetency milestone reporting form (see2). In the form, for each subcompetency, there are 10 response buttons arranged beneath 6 columns, each describing a set of milestone behaviors. The first column indicates that the Am J Kidney Dis. 2015;66(1):15-22
subcompetency is “not yet assessable,” that is, not observed. The second indicates that “critical deficiencies” exist. The next 3 columns describe satisfactory and progressively improving fellow performance. Column 3 describes an early learner (or noncritical deficiencies based on level of training), and column 4, a fellow who is satisfactorily progressing. Column 5, “ready for unsupervised practice,” is the graduation target. Column 6 describes “aspirational behavior.” To indicate an individual fellow’s progress, the CCC selects a response button either centered below a column or situated between columns. Selecting a button directly below a column indicates that milestones in that and preceding columns have been “substantially demonstrated,” whereas choosing a
From the Nephrology Service, Department of Medicine, Walter Reed National Military Medical Center, Bethesda, MD. Received October 10, 2014. Accepted in revised form January 8, 2015. Originally published online March 13, 2015. Address correspondence to Christina M. Yuan, MD, Nephrology SVC, Department of Medicine, Walter Reed National Military Medical Center, 8901 Wisconsin Ave, Bethesda, MD 20814. E-mail: [email protected] Published by Elsevier Inc. on behalf of the National Kidney Foundation, Inc. This is a US Government Work. There are no restrictions on its use. 0272-6386 http://dx.doi.org/10.1053/j.ajkd.2015.01.020 15
Yuan et al Box 1. ACGME Context-Nonspecific Internal Medicine Subspecialty Competencies and Subcompetencies Patient Care (PC) PC1: Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s). PC2: Develops and achieves comprehensive management plan for each patient. PC3: Manages patients with progressive responsibility and independence. PC4a: Demonstrates skill in performing and interpreting invasive procedures. PC4b: Demonstrates skill in performing and interpreting noninvasive procedures and/or testing. PC5: Requests and provides consultative care. Medical Knowledge (MK) MK1: Possesses clinical knowledge. MK2: Knowledge of diagnostic testing and procedures. MK3: Scholarship. Systems-Based Practice (SBP) SBP1: Works effectively within an interprofessional team (eg, with peers, consultants, nursing, ancillary professionals, and other support personnel). SBP2: Recognizes system error and advocates for system improvement. SBP3: Identifies forces that impact the cost of health care and advocates for and practices cost-effective care. SBP4: Transitions patients effectively within and across health delivery systems. Practice-Based Learning and Improvement (PBLI) PBLI1: Monitors practice with a goal for improvement. PBLI2: Learns and improves via performance audit. PBLI3: Learns and improves via feedback. PBLI4: Learns and improves at the point of care. Professionalism (PROF) PROF1: Has professional and respectful interactions with patients, caregivers, and members of the interprofessional team. PROF2: Accepts responsibility and follows through on tasks. PROF3: Responds to each patient’s unique characteristics and needs. PROF4: Exhibits integrity and ethical behavior in professional conduct. Interpersonal Communication Skills (ICS) ICS1: Communicates effectively with patients and caregivers. ICS2: Communicates effectively in interprofessional teams. ICS3: Appropriate utilization and completion of health records. Abbreviation: ACGME, Accreditation Council for Graduate Medical Education. Based on2 (which contains complete descriptions of subcompetency milestones).
button between columns indicates that milestones in preceding columns, as well as some in higher column(s), have been substantially demonstrated. After milestone attainment in each subcompetency has been assessed and documented, the CCC provides a summary evaluation of general competency performance.
NEPHROLOGY CURRICULAR MILESTONES: GOALS AND CHALLENGES Because the subspecialty milestone reporting template is nonspecific, the ACGME has tasked program directors and subspecialty societies with drafting specific curricular milestones in the context of the subspecialty.3 The American Society of Nephrology (ASN) Training Program Directors Working Committee released an implementation draft of nephrology curricular milestones and a compliance guide with assessment tools in June 2014.4,5 ACGME reporting began in late 2014. Thus, nephrology training programs were expected to have a structure for assessing and documenting milestone achievement in the 23 subcompetencies within 6 months of the implementation draft. 16
Among the worthy goals of the ACGME Milestones Project is a standardized framework for program accreditation decisions and publicly accountable continuous improvement in graduate medical education. For individual programs, milestones should provide curriculum development guidance, trainee assessment, feedback (with earlier identification of struggling trainees), and “explicit and transparent expectations of performance.”6 A critical limitation of the Milestones Project is the lack of clearly defined objective outcomes and validated assessment tools. Nephrology CCCs will determine trainee milestone achievement using the ASN draft curricular milestones and/or their own schema, arriving at consensus using nonstandardized evaluation forms and checklists, personal observations, and institution-specific assessment tools. They then will translate these assessments into context-nonspecific ACGME subspecialty milestones, aggregated and used by the ACGME to assess training program success in meeting yet-to-be-specified outcomes. This potentially chaotic approach is almost certain to produce biased and inaccurate information that is unlikely to yield actionable data on program or trainee success. Am J Kidney Dis. 2015;66(1):15-22
Nephrology Subspecialty Curricular Milestones
NEPHROLOGY CURRICULAR MILESTONES: THE EXPERIENCE AT WALTER REED Overview Previously, we described nephrology-specific milestones and objective assessment tools developed and used by our program.7 Subsequently, we described 2 assessment tools to determine milestone achievement: an objective structured clinical examination of dialysis emergency response and nephrology and transplantation clinic management as an entrustable professional activity (EPA) with performance assessed by a validated chart audit tool.8,9 Here, we map the 23 ACGME subspecialty subcompetency milestones to our nephrology-specific curricular milestone schema (Table 1) and describe the EPAs and associated assessment methods that inform our milestone decisions (Table 2). We have translated our schema into the ACGME subspecialty milestone-reporting format (Item S1) to allow comparison with the nonspecific ACGME subspecialty milestones and the curricular milestones developed by the ASN Program Directors.4 CCCs may convert our schema into the ACGME format with minimal additional effort. The objective assessment tools described in Tables 1 and 2 are supplemented by competencybased faculty evaluations of trainees after each rotation, developed by our program. Our faculty evaluations address nephrology-specific skills, such as provision of renal replacement therapy, kidney transplant management, and performance of kidney biopsy and temporary dialysis catheter placement. Our evaluation forms use a 5-point rating scheme that centers satisfactory performance for level of training at 3, inherently adjusting for progressive improvement over time and requiring written explanations for higher or lower ratings. A similar evaluation is used by peers and faculty for fellow academic presentations. Since our original report,7 we have added several milestones and assessment tools to better determine fellows’ progress toward the “ready for unsupervised practice” target (Table 1). Management of maintenance dialysis patients (an EPA; Table 2) is assessed using a monthly summary audit tool similar to that used for outpatient clinic encounters. The audit tool incorporates quality metrics required by the Centers for Medicare & Medicaid Services (eg, dialysis adequacy, anemia management, and dialysis access type) and informs 16 subcompetencies.9,10 Urinalysis competency (PC4b; Table 1) is assessed more fully by requiring first-year fellows to complete a test that complies with the College of American Pathologists point-of-care testing requirements, as well as an online urinalysis tutorial (see11). Competence in placing
Am J Kidney Dis. 2015;66(1):15-22
temporary dialysis catheters (PC4a; Table 1) is emphasized more strongly by giving a temporary dialysis catheter training simulation and checklist assessment during the first year of fellowship.12 We are developing an OSCE to assess trainee ability to appropriately select dialysis modality and write orders for acute kidney replacement therapy. Milestones for Clinical Knowledge Subcompetency Here, we use the MK1 (“possesses clinical knowledge”) subcompetency to demonstrate our approach. The ACGME nonspecific description of milestone progression in this subcompetency is subject to interpretation and potential evaluator bias.2 A fellow “ready for unsupervised practice” is described as possessing “the scientific, socioeconomic, and behavioral knowledge required to provide care for complex medical conditions and comprehensive preventive care.” A fellow who is satisfactorily progressing provides “care for common medical conditions and basic preventive care,” whereas an early learner/below-average fellow has “insufficient.knowledge required to provide care for common medical conditions and basic preventive care.” The ASN draft curricular milestones for MK1 focus exclusively on the attainment of one fairly narrow area of clinical knowledge in nephrology: the evaluation and selection of kidney transplant donors and recipients.4 Milestone attainment in this one area is used as an exemplar of MK1 milestone attainment generally. Our schema defines the MK1 subcompetency curricular milestones (Box 2) using criteria that are reproducible and objective, and provide clear goals for trainees who may not be progressing satisfactorily. Threshold results of ongoing general nephrology and transplantation clinic chart audits,7,9 end-of-month dialysis summaries audit, academic presentation evaluations, and faculty evaluations are used to make milestone determinations. Practical Example 1 A hypothetical fellow (Dr A) has the following CCC assessment of MK1 milestone achievement at 12 months into training. Chart audit of nephrology and transplantation clinic encounters (“manages general nephrology and transplant outpatient clinic” EPA) shows progressive improvement, and he achieved a monthly deficiency rate , 5% during the last 6 months. He completed his maintenance dialysis rotation (“manages panel of maintenance dialysis patients” EPA), and end-of-month summaries show a progressive decline in deficiencies, with .90% success at meeting quality indicators. He has given 4 formal academic presentations (“prepares and presents academic presentations and literature reviews”
17
Yuan et al Table 1. WRNMMC Nephrology Curricular Milestones Mapped to ACGME General Internal Medicine Subspecialty Milestones
Competency
PC
MK
PBLI
ICS
Fellowship Year-1 Milestones
ACGME Subcompetencya
Progressive decline in chart audit deficiencies (,5% at 6 mo)
PC1, PC2, PC3, PC5
Monthly dialysis summary chart audits demonstrate progressive decline in deficiencies with dialysis quality indicators met by end of training year Mini-CEX RRT and kidney biopsy counseling: progressive improvement Urinalysis competency and online tutorial Dialysis Emergencies OSCE: successful completion Temporary dialysis catheter placement simulation: successful checklist completion Progressive improvement in evaluations of prepared academic lectures
PC1, PC2, PC3, PC4a, PC5
Progressive decline in chart audit deficiencies (,5% at 6 mo)
MK1, MK2
Monthly dialysis summary chart audits demonstrate progressive decline in deficiencies with dialysis quality indicators met by end of training year Meets quality assurance metrics threshold on chart audit (eg, HTN management, identification and management of proteinuria; ,2% deficiency at 6 mo) Meets quality assurance metrics threshold on dialysis summary chart audit (eg, dialysis adequacy, anemia management; .90% of patients meet by end of first year) Increasing percentage of “yes” responses to the question “Will this presentation change your practice?” when applicable
MK1, MK2
PC4a
PC4b
Fellowship Year-2 Milestones
Reduction in 100% outpatient chart audit (mo 1-6) to 50% (mo 7-9) to 25% (mo 10-12), based on continued deficiency reduction (,5%) Threshold no. of successful procedures (kidney biopsy and temporary dialysis catheter placement) met
ACGME Subcompetency
PC1, PC2, PC3, PC5
PC4a
Mini-CEX RRT and kidney biopsy counseling: fully successful performance Urinalysis OSCE: successful completion
PC4a
Lecture content and presentation at faculty level, prepared independently, during the last half of the year Reduction in 100% outpatient chart audit (mo 1-6) to 50% (mo 7-9) to 25% (mo 10-12), based on continued deficiency reduction (,5%) Preparation of independent research protocol; publication/presentation of abstract, case report, or journal article
MK1, MK3
Meets quality assurance metrics threshold on chart-audit (eg, HTN management, identification and management of proteinuria; ,2% deficiency at 6 mo) Successful completion and presentation of nephrology-related, collaborative, multidisciplinary performance improvement project
PBLI1, PBLI2, PBLI3, PBLI4
PC4b
PC1, PC2, PC4a PC4a
MK1, MK3
PBLI1, PBLI2, PBLI3, PBLI4
PBLI1, PBLI2, PBLI3, PBLI4
PBLI1, PBLI3, PBLI4
Mini-CEX: scores progressively improve
ICS1
Chart audit: progressive improvement in timeliness and consultation skills Progressive improvement to satisfactory level by end of first year on 360 evaluations
ICS2, ICS3
ICS1, ICS2
.50% “yes” responses to the question “Will this presentation change your practice?” when applicable by the second half of the training year Successfully rated ($3) patient management conference assessing evidence-based best practice for patient seen in clinic or inpatient service Mini-CEX: scores progressively improve and are all fully acceptable by the end of the training year Chart audit: fully acceptable in timeliness and consultation skills Faculty level performance by the second half of the second year on 360 evaluations
MK1, MK2
MK3
PBLI2
PBLI1, PBLI3, PBLI4
PBLI4
ICS1
ICS2, ICS3
ICS1, ICS2
(Continued)
18
Am J Kidney Dis. 2015;66(1):15-22
Nephrology Subspecialty Curricular Milestones Table 1 (Cont’d). WRNMMC Nephrology Curricular Milestones Mapped to ACGME General Internal Medicine Subspecialty Milestones
Competency
PROF
SBP
Fellowship Year-1 Milestones
ACGME Subcompetencya
Progressive improvement to satisfactory performance on 360 evaluations Mini-CEX: scores progressively improve
PROF1, PROF2, PROF3, PROF4 PROF1, PROF3
Letters of praise from patients and praise from interprofessional team (not required) Progressive decline in chart audit deficiencies for timeliness of encounter completion (,5% at 6 mo)
PROF1, PROF2, PROF3, PROF4 PROF2
Proper and timely completion of CMS Form 2728
SBP1, SBP3, SBP4
Progressively improved military administrative actions related to kidney disease Progressive improvement to satisfactory level by end of first year on 360 evaluations
SBP4
SBP1
Fellowship Year-2 Milestones
Faculty level performance on 360 evaluations by the second half of the year Mini-CEX: scores progressively improve and are all fully acceptable by the end of the training year Letters of praise from patients and praise from interprofessional team (not required) Maintains deficiencies in timeliness of encounter completion to ,5% during last 12 mo Successful completion of medical ethics curriculum and exercises Correct completion of all military administrative actions related to kidney disease (,5% error by end of second year) Completion and presentation of mentored, multidisciplinary PI project. Faculty-level performance by the second half of second year on 360 evaluations
ACGME Subcompetency
PROF1, PROF2, PROF3, PROF4 PROF1, PROF 3
PROF1, PROF2, PROF3, PROF4 PROF2
PROF1, PROF3, PROF4 SBP4
SBP1, SBP2, SBP3 SBP1
Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; CEX, clinical evaluation exercise; CMS, Centers for Medicare & Medicaid Services; HTN, hypertension; ICS, Interpersonal Communication Skills; MK, Medical Knowledge; OSCE, objective structured clinical examination; PBLI, Practice-Based Learning and Improvement; PC, Patient Care; PI, performance improvement; PROF, Professionalism; RRT, renal replacement therapy; SBP, Systems-Based Practice; WRNMMC, Walter Reed National Military Medical Center. a For subcompetency descriptions, see Box 1.
EPA), and 37 evaluations show ratings of 3 (and sometimes 4) as he has progressed through the training year. Inpatient and outpatient/maintenance dialysis rotation evaluations show ratings of 3 (and sometimes 4) in MK, and the verbal assessment of CCC faculty members is consistent with these ratings. Thus, at the 1-year mark of a 2-year fellowship, Dr A has made “satisfactory progress for his level of training” in MK1, using our institution’s nephrology curricular milestone schema. For management of maintenance dialysis patients (EPA 2), he has reached the “ready for unsupervised practice” milestone, based in part on the CCC’s entrustment decision supported by objective performance on end-of-month dialysis summary audit. The CCC chooses the response button between column 4 and column 5 in the subspecialty milestone report form for MK1 (Item S1) because Dr A has fulfilled some, but not all, requirements for the “ready for unsupervised practice” milestone. Practical Example 2 A hypothetical fellow (Dr B) has the following CCC assessment of MK1 milestone achievement at 6 months into training. Chart audit of nephrology and Am J Kidney Dis. 2015;66(1):15-22
transplantation clinic encounters (“manages general nephrology and transplant outpatient clinic” EPA) show progressive improvement, but monthly deficiency rates were 7% last month, 8% the month before, and 9% the month before that. He often fails to meet quality indicators for hypertension management. Chronic kidney disease–associated metabolic bone disease management is confused and inconsistent. At 6 months into his maintenance dialysis rotation (“manages panel of maintenance dialysis patients” EPA), end-of-month summaries show a progressive decline in deficiencies, but only 75% success in meeting quality indicators for metabolic bone disease. After 2 formal academic presentations (“prepares and presents academic presentations and literature reviews” EPA), 15 evaluations show that his presentations generally receive ratings of 3. However, he received several ratings of 2 for his patient management conference, with comments that his presentation seemed hastily prepared and did not go beyond textbook information. Inpatient and outpatient/maintenance dialysis rotation MK evaluations show ratings of 2 and 3, and several faculty note that Dr B does not appear to be reading about glomerulonephritis cases 19
Yuan et al Table 2. ACGME General Subspecialty Subcompetencies Mapped to Nephrology-Specific Entrustable Professional Activities
Entrustable Professional Activity
Manages general nephrology and transplantation outpatient clinic Manages panel of maintenance dialysis patients Counsels patients for RRT and kidney biopsy Prepares and presents academic lectures and literature reviews Completes and presents a mentored multidisciplinary PI project
Evaluation Method
Evaluation by chart audit tool
Pertinent ACGME General Subspecialty Competenciesa
Evaluation by mini-CEX
PC1-3, 5; MK1, 2; SBP4; PBLI1-4; PROF2, 4; ICS2, 3 PC1-4a, 5; MK1, 2; SBP4; PBLI1-4; PROF2, 4; ICS2, 3 PC4a; MK2; PROF1, 3; ICS1
Conference presentation evaluations
MK1, 3; PBLI1, 3, 4, PROF2; ICS2
Completion and evaluation of presentation by nephrology faculty and staff
SBP1-3; PBLI1; PROF2; ICS2
Evaluation by dialysis summary chart audit tool
Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; CEX, clinical evaluation exercise; ICS, Interpersonal Communication Skills; MK, Medical Knowledge; PBLI, Practice-Based Learning and Improvement; PC, Patient Care; PI, performance improvement; PROF, Professionalism; RRT, renal replacement therapy; SBP, Systems-Based Practice. a For subcompetency descriptions, see Box 1.
or consistently developing a long-term immunosuppression plan for such patients. Verbal assessment of CCC faculty members is consistent with these observations. Thus, at the 6-month mark of a 2-year fellowship, Dr B falls between “satisfactory progress for level of training” and “below average progress for level of training” for MK1, and the CCC chooses the response button between columns 3 and 4 in the subspecialty milestone report form (Item S1). During formative counseling, the program director can point out specific deficiencies and develop a specific plan for remediation with clear objective thresholds for success or failure. In addition, the EPAs and assessment tools used to evaluate MK1 also inform the CCC that Dr B is exhibiting “below average progress” in other subcompetencies. For example, chart audit deficiencies associated with the “manages general nephrology and transplant outpatient clinic” EPA and faculty evaluations suggest that he has deficiencies in PC2 (“develops a comprehensive management plan for each patient”). His failure to meet quality indicators for hypertension and metabolic bone disease in outpatient clinic and maintenance dialysis patients suggests deficiencies in PBLI1 and PBLI2. Visualization of Progress If desired, the CCC can graphically demonstrate a fellow’s progress within each subcompetency using a variation of the radar graph, or “target chart.”13 In this version, successful performance (for which “aspirational” is scored as 5, and “ready for unsupervised practice,” as 4) is placed at the center of the chart, and deficient performance (“critical deficiencies,” scored as 1) is placed at the periphery. Thus, as the fellow successfully progresses through training, the area of his assessment grouping will decrease toward the center, or “target.” Figure S1 shows the MK1 target 20
charts for both fellows. Dr B, who is demonstrating below-average progress, has a larger area grouping than Dr A.
NEPHROLOGY CURRICULAR MILESTONES: IMPLEMENTATION CONSIDERATIONS We successfully used our curricular milestones during the first reporting period in late 2014. Because the ACGME will use aggregate fellow milestone performance to assess the quality of individual nephrology fellowship programs,6 we believe it is vital that reports be standardized, reproducible, and substantially objective. It is not clear what outcomes the ACGME will use to validate the milestones. Being found “ready for unsupervised practice” in each subcompetency at the conclusion of fellowship likely is the overwhelmingly common outcome of fellowship training. It is not yet clear how milestone achievement will intersect with nephrology subspecialty board examination eligibility; however, satisfactory performance in all competencies has been a requirement of the American Board of Internal Medicine (ABIM). The ABIM nephrology examination primarily assesses medical knowledge and is unlikely to effectively gauge readiness for unsupervised practice in the other 5 competencies. Although the ACGME subcompetency milestones provide a general, descriptive, and developmental framework, training programs and subspecialty societies are left with the daunting task of developing curricular milestones and assessment tools that accurately measure milestone attainment. It is not clear why the ACGME was in such haste to implement subspecialty milestone reporting in the 2014 to 2015 training year, not permitting sufficient time for the complex task of developing subspecialty consensus on curricular milestones and assessment tools. Am J Kidney Dis. 2015;66(1):15-22
Nephrology Subspecialty Curricular Milestones Box 2. Milestones for “Possesses Clinical Knowledge” Subcompetency (MK1) Not Yet Assessable Critical Deficiencies Academic lectures poor or not prepared Chart audit deficiencies . 5%, serious deficiencies, and no improvement First year: monthly dialysis summary chart audits with high deficiency rate, serious deficiencies, and no improvement Rating of 1 in MK on faculty evaluation Below-Average Progress for Level of Training Academic lectures not improving Chart audit deficiencies . 5%, but improving First year: monthly dialysis summary chart audits not showing decline in deficiencies Rating of 2 in MK on faculty evaluation Satisfactory Progress for Level of Training Progressive improvement in evaluations of prepared academic lectures Progressive decline in chart audit deficiencies (,5% at 6 mo) First year: monthly dialysis summary chart audits demonstrate progressive decline in deficiencies Rating of 3 in MK on faculty evaluation prior to last 6 months of training Ready for Unsupervised Practice Lecture content and presentation at faculty level, prepared independently, during the last half of the second year Second year: reduction in 100% outpatient chart audit (months 1-6) to 50% (months 7-9) to 25% (months 10-12), based on continued deficiency reduction (,5%) First year: monthly dialysis summary chart audits demonstrate progressive decline in deficiencies with dialysis quality indicators met by end of training year Rating $ 3 in MK on faculty evaluation during last 6 months of training Aspirational As determined by CCC (explanation to follow) Note: Pertinent assessment tools: 3 entrustable professional activities (“manages general nephrology and transplant outpatient clinic” [assessed by chart audit tool], “manages panel of maintenance dialysis patients” [assessed by chart audit tool], and “prepares and presents academic lectures and literature reviews” [assessed by conference evaluations]) and faculty rotation evaluations. Abbreviations: CCC, clinical competency committee; MK, medical knowledge.
Few validated assessment tools exist to inform the nephrology subspecialty milestones. The ASN nephrology curricular milestones in some subcompetencies (eg, PC4a, PC4b, SBP, and PBLI) are extremely detailed descriptions of behavioral progression.4 Some, such as PC1 and MK1, describe the evolution of a single representative skill (ability to assess volume status and ability to evaluate transplant donors and recipients, respectively), while others (PROF and ICS) are identical to the ACGME nonspecific Am J Kidney Dis. 2015;66(1):15-22
subcompetency milestones. Assessment tools offered by the ASN are largely traditional faculty and 360 evaluation forms and checklists, and individual CCCs are left to determine what evaluation score serves as a milestone attainment threshold.5 The temporary dialysis catheter simulation evaluation checklist of Barsuk et al12 also is offered, but not in the context of simulation training. The phenomenologic 5-stage model of skill acquisition of Dreyfus and Dreyfus14 is a source for both the internal medicine and the context-nonspecific subspecialty milestones.15 The milestones reporting tool references the Dreyfus model in its use of the 5 progressive states of competence. This model has been applied to medical education, particularly in the acquisition of procedural and nursing skills.16,17 However, it has been criticized for oversimplifying the acquisition of diagnostic and clinical problemsolving skills. This may be especially pertinent for subspecialty fellows, who would not be considered novice learners within the Dreyfus framework because they are already trained internists and unlikely to approach subspecialty material in a contextfree rule-based way.18 The Dreyfus model does not incorporate assessment recommendations, and the weakness of ACGME competency-based subspecialty milestones is that they lack clearly defined objective outcomes and validated assessment tools.19,20 The ABIM nephrology examination, a possible outcome, primarily assesses medical knowledge. The ACGME may hope to leverage large-scale electronic health record data to ultimately associate patient outcomes with training program performance.21 This approach might be problematic if data quality for aggregate milestone attainment is poor, especially if the approach is biased by the imperative to find nearly all trainees “ready for unsupervised practice” at the end of training. Milestone attainment data are likely to be biased and heterogeneous, especially if based on unstandardized subjective descriptions of fellow behavior. Data harmonization (ie, shared terminology and expectations) is critical. Potential confounders and assumptions about causal relationships between milestone attainment and trainee/patient outcomes must be articulated prospectively to avoid spurious correlations. Such analyses are complex and require sophisticated, transparent, peer-reviewed statistical methods. A rigorous process is particularly necessary considering the serious consequences to training programs that may be identified as poorly performing. Moreover, retrospective observational association of trainee or patient outcomes with individual training programs may not be able to address causality because of systematic bias.22,23 21
Yuan et al
Our nephrology curricular milestones incorporate outcome thresholds derived from objective assessment tools, rather than exclusively depending on descriptions of fellow behavior. We believe they offer a less biased and more efficient and reproducible framework for determining milestone attainment. Ultimately, nephrology training programs must agree on a standardized set of curricular milestones and develop measurable, objective, validated milestone assessment tools. These must be transparent and submitted to peer review, as must ACGME analyses that use aggregated milestone achievement data. If useful data are to be produced regarding effective nephrology training tools, collaborative educational research investments (including financial investments) will be required.
ACKNOWLEDGEMENTS The views expressed in this report are those of the authors and do not reflect the official policy of the Department of the Army, the Department of the Navy, the Department of Defense, or the United States Government. Support: None. Financial Disclosure: The authors declare that they have no relevant financial interests.
SUPPLEMENTARY MATERIAL Figure S1: Sample target charts. Item S1: Walter Reed National Military Medical Center Nephrology Curricular Subcompetency Milestones (including MK1). Note: The supplementary material accompanying this article (http://dx.doi.org/10.1053/j.ajkd.2015.01.020) is available at www.ajkd.org
REFERENCES 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366:1051-1056. 2. The Internal Medicine Subspecialty Milestones Project: a joint initiative of the Accreditation Council for Graduate Medical Education and the American Board of Internal Medicine, in collaboration with the Alliance for Academic Internal Medicine. October 2014. http://acgme.org/acgmeweb/Portals/0/PDFs/Milestones/Internal MedicineSubspecialtyMilestones.pdf. Accessed December 16, 2014. 3. Vasilias J, Arrighi JA. Residency Review Committee for Internal Medicine (RRC-IM) update. APDIM fall meeting, October 2013. https://www.acgme.org/acgmeweb/Portals/0/ PFAssets/Presentations/140_APDIM_Oct2013_RCIM_Update.pdf. Accessed March 1, 2014. 4. Nephrology Curricular Milestones: implementation draft. American Society of Nephrology Training Program Directors. May 2014. https://www.asn-online.org/education/training/tpd/ ASN_Nephrology_Curricular_Milestones.pdf Accessed August 28, 2014. 5. American Society of Nephrology Training Program Directors Toolkit. https://www.asn-online.org/education/training/tpd/ toolkit.aspx. Accessed September 2, 2014.
22
6. Next Accreditation System: milestones. The Accreditation Council for Graduate Medical Education. http://www.acgme.org/ acgmeweb/tabid/430/ProgramandInstitutionalAccreditation/Next AccreditationSystem/Milestones.aspx. Accessed December 26, 2014. 7. Yuan CM, Nee R, Abbott KC, Oliver JD, III. Milestones for nephrology training programs: a modest proposal. Am J Kidney Dis. 2013;62(6):1034-1038. 8. Prince LK, Abbott KC, Green F, et al. Expanding the role of objectively structured clinical examinations in nephrology training. Am J Kidney Dis. 2014;63(6):906-912. 9. Yuan CM, Prince LK, Zwettler AJ, Nee R, Oliver JD III, Abbott KC. Assessing achievement in nephrology training: using clinic chart audits to quantitatively screen competency. Am J Kidney Dis. 2014;64(5):737-743. 10. Centers for Medicare and Medicaid Services (CMS). ESRD QIP summary: payment years 2012-2016. http://www.cms. gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/ ESRDQIP/Downloads/ESRD-QIP-Sumary-PY2012-16.pdf. Accessed September 2, 2014. 11. University of Washington Department of Laboratory Medicine. Online training and competency assessment for the clinical lab and point of care. http://www.medtraining.org. Accessed February 14, 2015. 12. Barsuk JH, Ahya SN, Cohen ER, McGaghie WC, Wayne DB. Mastery learning of temporary hemodialysis catheter insertion by nephrology fellows using simulation technology and deliberate practice. Am J Kidney Dis. 2009;54(1):70-76. 13. Keister DM, Larson D, Dostal J, Baglia J. The radar graph: the development of an educational tool to demonstrate resident competency. J Grad Med Educ. 2012;4(2):220-226. 14. Dreyfus H, Drefus SE. Mind over machine. In: Five Steps From Novice to Expert. New York, NY: Free Press; 1986:16-51. 15. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: developmental milestones for internal medicine residency training. J Grad Med Educ. 2009;1(1):5-20. 16. Aggarwal R, Darzi LA. Measurement of surgical performance for delivery of a competency-based training curriculum. In: Athanasoiu T, Debas H, Darzi A, eds. Key Topics in Surgical Research and Methodology. Berlin, Germany: Springer-Verlag; 2010:115-126. 17. Benner P. Using the Dreyfus model of skill acquisition to describe and interpret skill acquisition and clinical judgment in nursing practice and education. Bull Sci Technol Soc. 2004;24(3):188-199. 18. Pena A. The Dreyfus model of clinical problem-solving skills acquisition: a critical perspective. Med Educ Online. 2010;15:1-11. http://dx.doi.org/10.3402/meo.v15i0.4846. 19. Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: translating the Dreyfus developmental model to the learning of clinical skills. Acad Med. 2008;83(8):761-767. 20. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682. 21. Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ. Evaluating obstetrical residency programs using patient outcomes. JAMA. 2009;302(12):1277-1283. 22. Hoffman S, Podgurski A. Big bad data: law, public health, and biomedical databases. J Law Med Ethics. 2013;41(suppl 1):56-60. 23. Fan J, Han F, Liu H. Challenges of big data analysis. Natl Sci Rev. 2014;193:293-314.
Am J Kidney Dis. 2015;66(1):15-22
I
n 2012, the Accreditation Council for Graduate Medical Education (ACGME) introduced the Next Accreditation System.1 One objective of this system is to base training program accreditation on trainee outcomes, using milestone attainment in the 6 basic competencies: Patient Care (PC), Medical Knowledge (MK), Professionalism (PROF), Interpersonal Communication Skills (ICS), Practice-Based Learning and Improvement (PBLI), and SystemsBased Practice (SBP). Milestones are competencybased progressive achievements, demonstrated by trainees during the period of their education and culminating in graduation and the commencement of unsupervised practice. Aggregate performance of program trainees in achieving milestones will be one outcome data element used annually by the Next Accreditation System to assess programmatic success/ compliance, ultimately producing national competency outcomes within specialties and subspecialties. In March 2014, the ACGME published “context nonspecific” internal medicine subspecialty milestones in 23 subcompetencies (Box 1).2 Clinical Competency Committees (CCCs) will submit twice-yearly trainee evaluations using the subcompetency milestone reporting form (see2). In the form, for each subcompetency, there are 10 response buttons arranged beneath 6 columns, each describing a set of milestone behaviors. The first column indicates that the Am J Kidney Dis. 2015;66(1):15-22
subcompetency is “not yet assessable,” that is, not observed. The second indicates that “critical deficiencies” exist. The next 3 columns describe satisfactory and progressively improving fellow performance. Column 3 describes an early learner (or noncritical deficiencies based on level of training), and column 4, a fellow who is satisfactorily progressing. Column 5, “ready for unsupervised practice,” is the graduation target. Column 6 describes “aspirational behavior.” To indicate an individual fellow’s progress, the CCC selects a response button either centered below a column or situated between columns. Selecting a button directly below a column indicates that milestones in that and preceding columns have been “substantially demonstrated,” whereas choosing a
From the Nephrology Service, Department of Medicine, Walter Reed National Military Medical Center, Bethesda, MD. Received October 10, 2014. Accepted in revised form January 8, 2015. Originally published online March 13, 2015. Address correspondence to Christina M. Yuan, MD, Nephrology SVC, Department of Medicine, Walter Reed National Military Medical Center, 8901 Wisconsin Ave, Bethesda, MD 20814. E-mail: [email protected] Published by Elsevier Inc. on behalf of the National Kidney Foundation, Inc. This is a US Government Work. There are no restrictions on its use. 0272-6386 http://dx.doi.org/10.1053/j.ajkd.2015.01.020 15
Yuan et al Box 1. ACGME Context-Nonspecific Internal Medicine Subspecialty Competencies and Subcompetencies Patient Care (PC) PC1: Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s). PC2: Develops and achieves comprehensive management plan for each patient. PC3: Manages patients with progressive responsibility and independence. PC4a: Demonstrates skill in performing and interpreting invasive procedures. PC4b: Demonstrates skill in performing and interpreting noninvasive procedures and/or testing. PC5: Requests and provides consultative care. Medical Knowledge (MK) MK1: Possesses clinical knowledge. MK2: Knowledge of diagnostic testing and procedures. MK3: Scholarship. Systems-Based Practice (SBP) SBP1: Works effectively within an interprofessional team (eg, with peers, consultants, nursing, ancillary professionals, and other support personnel). SBP2: Recognizes system error and advocates for system improvement. SBP3: Identifies forces that impact the cost of health care and advocates for and practices cost-effective care. SBP4: Transitions patients effectively within and across health delivery systems. Practice-Based Learning and Improvement (PBLI) PBLI1: Monitors practice with a goal for improvement. PBLI2: Learns and improves via performance audit. PBLI3: Learns and improves via feedback. PBLI4: Learns and improves at the point of care. Professionalism (PROF) PROF1: Has professional and respectful interactions with patients, caregivers, and members of the interprofessional team. PROF2: Accepts responsibility and follows through on tasks. PROF3: Responds to each patient’s unique characteristics and needs. PROF4: Exhibits integrity and ethical behavior in professional conduct. Interpersonal Communication Skills (ICS) ICS1: Communicates effectively with patients and caregivers. ICS2: Communicates effectively in interprofessional teams. ICS3: Appropriate utilization and completion of health records. Abbreviation: ACGME, Accreditation Council for Graduate Medical Education. Based on2 (which contains complete descriptions of subcompetency milestones).
button between columns indicates that milestones in preceding columns, as well as some in higher column(s), have been substantially demonstrated. After milestone attainment in each subcompetency has been assessed and documented, the CCC provides a summary evaluation of general competency performance.
NEPHROLOGY CURRICULAR MILESTONES: GOALS AND CHALLENGES Because the subspecialty milestone reporting template is nonspecific, the ACGME has tasked program directors and subspecialty societies with drafting specific curricular milestones in the context of the subspecialty.3 The American Society of Nephrology (ASN) Training Program Directors Working Committee released an implementation draft of nephrology curricular milestones and a compliance guide with assessment tools in June 2014.4,5 ACGME reporting began in late 2014. Thus, nephrology training programs were expected to have a structure for assessing and documenting milestone achievement in the 23 subcompetencies within 6 months of the implementation draft. 16
Among the worthy goals of the ACGME Milestones Project is a standardized framework for program accreditation decisions and publicly accountable continuous improvement in graduate medical education. For individual programs, milestones should provide curriculum development guidance, trainee assessment, feedback (with earlier identification of struggling trainees), and “explicit and transparent expectations of performance.”6 A critical limitation of the Milestones Project is the lack of clearly defined objective outcomes and validated assessment tools. Nephrology CCCs will determine trainee milestone achievement using the ASN draft curricular milestones and/or their own schema, arriving at consensus using nonstandardized evaluation forms and checklists, personal observations, and institution-specific assessment tools. They then will translate these assessments into context-nonspecific ACGME subspecialty milestones, aggregated and used by the ACGME to assess training program success in meeting yet-to-be-specified outcomes. This potentially chaotic approach is almost certain to produce biased and inaccurate information that is unlikely to yield actionable data on program or trainee success. Am J Kidney Dis. 2015;66(1):15-22
Nephrology Subspecialty Curricular Milestones
NEPHROLOGY CURRICULAR MILESTONES: THE EXPERIENCE AT WALTER REED Overview Previously, we described nephrology-specific milestones and objective assessment tools developed and used by our program.7 Subsequently, we described 2 assessment tools to determine milestone achievement: an objective structured clinical examination of dialysis emergency response and nephrology and transplantation clinic management as an entrustable professional activity (EPA) with performance assessed by a validated chart audit tool.8,9 Here, we map the 23 ACGME subspecialty subcompetency milestones to our nephrology-specific curricular milestone schema (Table 1) and describe the EPAs and associated assessment methods that inform our milestone decisions (Table 2). We have translated our schema into the ACGME subspecialty milestone-reporting format (Item S1) to allow comparison with the nonspecific ACGME subspecialty milestones and the curricular milestones developed by the ASN Program Directors.4 CCCs may convert our schema into the ACGME format with minimal additional effort. The objective assessment tools described in Tables 1 and 2 are supplemented by competencybased faculty evaluations of trainees after each rotation, developed by our program. Our faculty evaluations address nephrology-specific skills, such as provision of renal replacement therapy, kidney transplant management, and performance of kidney biopsy and temporary dialysis catheter placement. Our evaluation forms use a 5-point rating scheme that centers satisfactory performance for level of training at 3, inherently adjusting for progressive improvement over time and requiring written explanations for higher or lower ratings. A similar evaluation is used by peers and faculty for fellow academic presentations. Since our original report,7 we have added several milestones and assessment tools to better determine fellows’ progress toward the “ready for unsupervised practice” target (Table 1). Management of maintenance dialysis patients (an EPA; Table 2) is assessed using a monthly summary audit tool similar to that used for outpatient clinic encounters. The audit tool incorporates quality metrics required by the Centers for Medicare & Medicaid Services (eg, dialysis adequacy, anemia management, and dialysis access type) and informs 16 subcompetencies.9,10 Urinalysis competency (PC4b; Table 1) is assessed more fully by requiring first-year fellows to complete a test that complies with the College of American Pathologists point-of-care testing requirements, as well as an online urinalysis tutorial (see11). Competence in placing
Am J Kidney Dis. 2015;66(1):15-22
temporary dialysis catheters (PC4a; Table 1) is emphasized more strongly by giving a temporary dialysis catheter training simulation and checklist assessment during the first year of fellowship.12 We are developing an OSCE to assess trainee ability to appropriately select dialysis modality and write orders for acute kidney replacement therapy. Milestones for Clinical Knowledge Subcompetency Here, we use the MK1 (“possesses clinical knowledge”) subcompetency to demonstrate our approach. The ACGME nonspecific description of milestone progression in this subcompetency is subject to interpretation and potential evaluator bias.2 A fellow “ready for unsupervised practice” is described as possessing “the scientific, socioeconomic, and behavioral knowledge required to provide care for complex medical conditions and comprehensive preventive care.” A fellow who is satisfactorily progressing provides “care for common medical conditions and basic preventive care,” whereas an early learner/below-average fellow has “insufficient.knowledge required to provide care for common medical conditions and basic preventive care.” The ASN draft curricular milestones for MK1 focus exclusively on the attainment of one fairly narrow area of clinical knowledge in nephrology: the evaluation and selection of kidney transplant donors and recipients.4 Milestone attainment in this one area is used as an exemplar of MK1 milestone attainment generally. Our schema defines the MK1 subcompetency curricular milestones (Box 2) using criteria that are reproducible and objective, and provide clear goals for trainees who may not be progressing satisfactorily. Threshold results of ongoing general nephrology and transplantation clinic chart audits,7,9 end-of-month dialysis summaries audit, academic presentation evaluations, and faculty evaluations are used to make milestone determinations. Practical Example 1 A hypothetical fellow (Dr A) has the following CCC assessment of MK1 milestone achievement at 12 months into training. Chart audit of nephrology and transplantation clinic encounters (“manages general nephrology and transplant outpatient clinic” EPA) shows progressive improvement, and he achieved a monthly deficiency rate , 5% during the last 6 months. He completed his maintenance dialysis rotation (“manages panel of maintenance dialysis patients” EPA), and end-of-month summaries show a progressive decline in deficiencies, with .90% success at meeting quality indicators. He has given 4 formal academic presentations (“prepares and presents academic presentations and literature reviews”
17
Yuan et al Table 1. WRNMMC Nephrology Curricular Milestones Mapped to ACGME General Internal Medicine Subspecialty Milestones
Competency
PC
MK
PBLI
ICS
Fellowship Year-1 Milestones
ACGME Subcompetencya
Progressive decline in chart audit deficiencies (,5% at 6 mo)
PC1, PC2, PC3, PC5
Monthly dialysis summary chart audits demonstrate progressive decline in deficiencies with dialysis quality indicators met by end of training year Mini-CEX RRT and kidney biopsy counseling: progressive improvement Urinalysis competency and online tutorial Dialysis Emergencies OSCE: successful completion Temporary dialysis catheter placement simulation: successful checklist completion Progressive improvement in evaluations of prepared academic lectures
PC1, PC2, PC3, PC4a, PC5
Progressive decline in chart audit deficiencies (,5% at 6 mo)
MK1, MK2
Monthly dialysis summary chart audits demonstrate progressive decline in deficiencies with dialysis quality indicators met by end of training year Meets quality assurance metrics threshold on chart audit (eg, HTN management, identification and management of proteinuria; ,2% deficiency at 6 mo) Meets quality assurance metrics threshold on dialysis summary chart audit (eg, dialysis adequacy, anemia management; .90% of patients meet by end of first year) Increasing percentage of “yes” responses to the question “Will this presentation change your practice?” when applicable
MK1, MK2
PC4a
PC4b
Fellowship Year-2 Milestones
Reduction in 100% outpatient chart audit (mo 1-6) to 50% (mo 7-9) to 25% (mo 10-12), based on continued deficiency reduction (,5%) Threshold no. of successful procedures (kidney biopsy and temporary dialysis catheter placement) met
ACGME Subcompetency
PC1, PC2, PC3, PC5
PC4a
Mini-CEX RRT and kidney biopsy counseling: fully successful performance Urinalysis OSCE: successful completion
PC4a
Lecture content and presentation at faculty level, prepared independently, during the last half of the year Reduction in 100% outpatient chart audit (mo 1-6) to 50% (mo 7-9) to 25% (mo 10-12), based on continued deficiency reduction (,5%) Preparation of independent research protocol; publication/presentation of abstract, case report, or journal article
MK1, MK3
Meets quality assurance metrics threshold on chart-audit (eg, HTN management, identification and management of proteinuria; ,2% deficiency at 6 mo) Successful completion and presentation of nephrology-related, collaborative, multidisciplinary performance improvement project
PBLI1, PBLI2, PBLI3, PBLI4
PC4b
PC1, PC2, PC4a PC4a
MK1, MK3
PBLI1, PBLI2, PBLI3, PBLI4
PBLI1, PBLI2, PBLI3, PBLI4
PBLI1, PBLI3, PBLI4
Mini-CEX: scores progressively improve
ICS1
Chart audit: progressive improvement in timeliness and consultation skills Progressive improvement to satisfactory level by end of first year on 360 evaluations
ICS2, ICS3
ICS1, ICS2
.50% “yes” responses to the question “Will this presentation change your practice?” when applicable by the second half of the training year Successfully rated ($3) patient management conference assessing evidence-based best practice for patient seen in clinic or inpatient service Mini-CEX: scores progressively improve and are all fully acceptable by the end of the training year Chart audit: fully acceptable in timeliness and consultation skills Faculty level performance by the second half of the second year on 360 evaluations
MK1, MK2
MK3
PBLI2
PBLI1, PBLI3, PBLI4
PBLI4
ICS1
ICS2, ICS3
ICS1, ICS2
(Continued)
18
Am J Kidney Dis. 2015;66(1):15-22
Nephrology Subspecialty Curricular Milestones Table 1 (Cont’d). WRNMMC Nephrology Curricular Milestones Mapped to ACGME General Internal Medicine Subspecialty Milestones
Competency
PROF
SBP
Fellowship Year-1 Milestones
ACGME Subcompetencya
Progressive improvement to satisfactory performance on 360 evaluations Mini-CEX: scores progressively improve
PROF1, PROF2, PROF3, PROF4 PROF1, PROF3
Letters of praise from patients and praise from interprofessional team (not required) Progressive decline in chart audit deficiencies for timeliness of encounter completion (,5% at 6 mo)
PROF1, PROF2, PROF3, PROF4 PROF2
Proper and timely completion of CMS Form 2728
SBP1, SBP3, SBP4
Progressively improved military administrative actions related to kidney disease Progressive improvement to satisfactory level by end of first year on 360 evaluations
SBP4
SBP1
Fellowship Year-2 Milestones
Faculty level performance on 360 evaluations by the second half of the year Mini-CEX: scores progressively improve and are all fully acceptable by the end of the training year Letters of praise from patients and praise from interprofessional team (not required) Maintains deficiencies in timeliness of encounter completion to ,5% during last 12 mo Successful completion of medical ethics curriculum and exercises Correct completion of all military administrative actions related to kidney disease (,5% error by end of second year) Completion and presentation of mentored, multidisciplinary PI project. Faculty-level performance by the second half of second year on 360 evaluations
ACGME Subcompetency
PROF1, PROF2, PROF3, PROF4 PROF1, PROF 3
PROF1, PROF2, PROF3, PROF4 PROF2
PROF1, PROF3, PROF4 SBP4
SBP1, SBP2, SBP3 SBP1
Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; CEX, clinical evaluation exercise; CMS, Centers for Medicare & Medicaid Services; HTN, hypertension; ICS, Interpersonal Communication Skills; MK, Medical Knowledge; OSCE, objective structured clinical examination; PBLI, Practice-Based Learning and Improvement; PC, Patient Care; PI, performance improvement; PROF, Professionalism; RRT, renal replacement therapy; SBP, Systems-Based Practice; WRNMMC, Walter Reed National Military Medical Center. a For subcompetency descriptions, see Box 1.
EPA), and 37 evaluations show ratings of 3 (and sometimes 4) as he has progressed through the training year. Inpatient and outpatient/maintenance dialysis rotation evaluations show ratings of 3 (and sometimes 4) in MK, and the verbal assessment of CCC faculty members is consistent with these ratings. Thus, at the 1-year mark of a 2-year fellowship, Dr A has made “satisfactory progress for his level of training” in MK1, using our institution’s nephrology curricular milestone schema. For management of maintenance dialysis patients (EPA 2), he has reached the “ready for unsupervised practice” milestone, based in part on the CCC’s entrustment decision supported by objective performance on end-of-month dialysis summary audit. The CCC chooses the response button between column 4 and column 5 in the subspecialty milestone report form for MK1 (Item S1) because Dr A has fulfilled some, but not all, requirements for the “ready for unsupervised practice” milestone. Practical Example 2 A hypothetical fellow (Dr B) has the following CCC assessment of MK1 milestone achievement at 6 months into training. Chart audit of nephrology and Am J Kidney Dis. 2015;66(1):15-22
transplantation clinic encounters (“manages general nephrology and transplant outpatient clinic” EPA) show progressive improvement, but monthly deficiency rates were 7% last month, 8% the month before, and 9% the month before that. He often fails to meet quality indicators for hypertension management. Chronic kidney disease–associated metabolic bone disease management is confused and inconsistent. At 6 months into his maintenance dialysis rotation (“manages panel of maintenance dialysis patients” EPA), end-of-month summaries show a progressive decline in deficiencies, but only 75% success in meeting quality indicators for metabolic bone disease. After 2 formal academic presentations (“prepares and presents academic presentations and literature reviews” EPA), 15 evaluations show that his presentations generally receive ratings of 3. However, he received several ratings of 2 for his patient management conference, with comments that his presentation seemed hastily prepared and did not go beyond textbook information. Inpatient and outpatient/maintenance dialysis rotation MK evaluations show ratings of 2 and 3, and several faculty note that Dr B does not appear to be reading about glomerulonephritis cases 19
Yuan et al Table 2. ACGME General Subspecialty Subcompetencies Mapped to Nephrology-Specific Entrustable Professional Activities
Entrustable Professional Activity
Manages general nephrology and transplantation outpatient clinic Manages panel of maintenance dialysis patients Counsels patients for RRT and kidney biopsy Prepares and presents academic lectures and literature reviews Completes and presents a mentored multidisciplinary PI project
Evaluation Method
Evaluation by chart audit tool
Pertinent ACGME General Subspecialty Competenciesa
Evaluation by mini-CEX
PC1-3, 5; MK1, 2; SBP4; PBLI1-4; PROF2, 4; ICS2, 3 PC1-4a, 5; MK1, 2; SBP4; PBLI1-4; PROF2, 4; ICS2, 3 PC4a; MK2; PROF1, 3; ICS1
Conference presentation evaluations
MK1, 3; PBLI1, 3, 4, PROF2; ICS2
Completion and evaluation of presentation by nephrology faculty and staff
SBP1-3; PBLI1; PROF2; ICS2
Evaluation by dialysis summary chart audit tool
Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; CEX, clinical evaluation exercise; ICS, Interpersonal Communication Skills; MK, Medical Knowledge; PBLI, Practice-Based Learning and Improvement; PC, Patient Care; PI, performance improvement; PROF, Professionalism; RRT, renal replacement therapy; SBP, Systems-Based Practice. a For subcompetency descriptions, see Box 1.
or consistently developing a long-term immunosuppression plan for such patients. Verbal assessment of CCC faculty members is consistent with these observations. Thus, at the 6-month mark of a 2-year fellowship, Dr B falls between “satisfactory progress for level of training” and “below average progress for level of training” for MK1, and the CCC chooses the response button between columns 3 and 4 in the subspecialty milestone report form (Item S1). During formative counseling, the program director can point out specific deficiencies and develop a specific plan for remediation with clear objective thresholds for success or failure. In addition, the EPAs and assessment tools used to evaluate MK1 also inform the CCC that Dr B is exhibiting “below average progress” in other subcompetencies. For example, chart audit deficiencies associated with the “manages general nephrology and transplant outpatient clinic” EPA and faculty evaluations suggest that he has deficiencies in PC2 (“develops a comprehensive management plan for each patient”). His failure to meet quality indicators for hypertension and metabolic bone disease in outpatient clinic and maintenance dialysis patients suggests deficiencies in PBLI1 and PBLI2. Visualization of Progress If desired, the CCC can graphically demonstrate a fellow’s progress within each subcompetency using a variation of the radar graph, or “target chart.”13 In this version, successful performance (for which “aspirational” is scored as 5, and “ready for unsupervised practice,” as 4) is placed at the center of the chart, and deficient performance (“critical deficiencies,” scored as 1) is placed at the periphery. Thus, as the fellow successfully progresses through training, the area of his assessment grouping will decrease toward the center, or “target.” Figure S1 shows the MK1 target 20
charts for both fellows. Dr B, who is demonstrating below-average progress, has a larger area grouping than Dr A.
NEPHROLOGY CURRICULAR MILESTONES: IMPLEMENTATION CONSIDERATIONS We successfully used our curricular milestones during the first reporting period in late 2014. Because the ACGME will use aggregate fellow milestone performance to assess the quality of individual nephrology fellowship programs,6 we believe it is vital that reports be standardized, reproducible, and substantially objective. It is not clear what outcomes the ACGME will use to validate the milestones. Being found “ready for unsupervised practice” in each subcompetency at the conclusion of fellowship likely is the overwhelmingly common outcome of fellowship training. It is not yet clear how milestone achievement will intersect with nephrology subspecialty board examination eligibility; however, satisfactory performance in all competencies has been a requirement of the American Board of Internal Medicine (ABIM). The ABIM nephrology examination primarily assesses medical knowledge and is unlikely to effectively gauge readiness for unsupervised practice in the other 5 competencies. Although the ACGME subcompetency milestones provide a general, descriptive, and developmental framework, training programs and subspecialty societies are left with the daunting task of developing curricular milestones and assessment tools that accurately measure milestone attainment. It is not clear why the ACGME was in such haste to implement subspecialty milestone reporting in the 2014 to 2015 training year, not permitting sufficient time for the complex task of developing subspecialty consensus on curricular milestones and assessment tools. Am J Kidney Dis. 2015;66(1):15-22
Nephrology Subspecialty Curricular Milestones Box 2. Milestones for “Possesses Clinical Knowledge” Subcompetency (MK1) Not Yet Assessable Critical Deficiencies Academic lectures poor or not prepared Chart audit deficiencies . 5%, serious deficiencies, and no improvement First year: monthly dialysis summary chart audits with high deficiency rate, serious deficiencies, and no improvement Rating of 1 in MK on faculty evaluation Below-Average Progress for Level of Training Academic lectures not improving Chart audit deficiencies . 5%, but improving First year: monthly dialysis summary chart audits not showing decline in deficiencies Rating of 2 in MK on faculty evaluation Satisfactory Progress for Level of Training Progressive improvement in evaluations of prepared academic lectures Progressive decline in chart audit deficiencies (,5% at 6 mo) First year: monthly dialysis summary chart audits demonstrate progressive decline in deficiencies Rating of 3 in MK on faculty evaluation prior to last 6 months of training Ready for Unsupervised Practice Lecture content and presentation at faculty level, prepared independently, during the last half of the second year Second year: reduction in 100% outpatient chart audit (months 1-6) to 50% (months 7-9) to 25% (months 10-12), based on continued deficiency reduction (,5%) First year: monthly dialysis summary chart audits demonstrate progressive decline in deficiencies with dialysis quality indicators met by end of training year Rating $ 3 in MK on faculty evaluation during last 6 months of training Aspirational As determined by CCC (explanation to follow) Note: Pertinent assessment tools: 3 entrustable professional activities (“manages general nephrology and transplant outpatient clinic” [assessed by chart audit tool], “manages panel of maintenance dialysis patients” [assessed by chart audit tool], and “prepares and presents academic lectures and literature reviews” [assessed by conference evaluations]) and faculty rotation evaluations. Abbreviations: CCC, clinical competency committee; MK, medical knowledge.
Few validated assessment tools exist to inform the nephrology subspecialty milestones. The ASN nephrology curricular milestones in some subcompetencies (eg, PC4a, PC4b, SBP, and PBLI) are extremely detailed descriptions of behavioral progression.4 Some, such as PC1 and MK1, describe the evolution of a single representative skill (ability to assess volume status and ability to evaluate transplant donors and recipients, respectively), while others (PROF and ICS) are identical to the ACGME nonspecific Am J Kidney Dis. 2015;66(1):15-22
subcompetency milestones. Assessment tools offered by the ASN are largely traditional faculty and 360 evaluation forms and checklists, and individual CCCs are left to determine what evaluation score serves as a milestone attainment threshold.5 The temporary dialysis catheter simulation evaluation checklist of Barsuk et al12 also is offered, but not in the context of simulation training. The phenomenologic 5-stage model of skill acquisition of Dreyfus and Dreyfus14 is a source for both the internal medicine and the context-nonspecific subspecialty milestones.15 The milestones reporting tool references the Dreyfus model in its use of the 5 progressive states of competence. This model has been applied to medical education, particularly in the acquisition of procedural and nursing skills.16,17 However, it has been criticized for oversimplifying the acquisition of diagnostic and clinical problemsolving skills. This may be especially pertinent for subspecialty fellows, who would not be considered novice learners within the Dreyfus framework because they are already trained internists and unlikely to approach subspecialty material in a contextfree rule-based way.18 The Dreyfus model does not incorporate assessment recommendations, and the weakness of ACGME competency-based subspecialty milestones is that they lack clearly defined objective outcomes and validated assessment tools.19,20 The ABIM nephrology examination, a possible outcome, primarily assesses medical knowledge. The ACGME may hope to leverage large-scale electronic health record data to ultimately associate patient outcomes with training program performance.21 This approach might be problematic if data quality for aggregate milestone attainment is poor, especially if the approach is biased by the imperative to find nearly all trainees “ready for unsupervised practice” at the end of training. Milestone attainment data are likely to be biased and heterogeneous, especially if based on unstandardized subjective descriptions of fellow behavior. Data harmonization (ie, shared terminology and expectations) is critical. Potential confounders and assumptions about causal relationships between milestone attainment and trainee/patient outcomes must be articulated prospectively to avoid spurious correlations. Such analyses are complex and require sophisticated, transparent, peer-reviewed statistical methods. A rigorous process is particularly necessary considering the serious consequences to training programs that may be identified as poorly performing. Moreover, retrospective observational association of trainee or patient outcomes with individual training programs may not be able to address causality because of systematic bias.22,23 21
Yuan et al
Our nephrology curricular milestones incorporate outcome thresholds derived from objective assessment tools, rather than exclusively depending on descriptions of fellow behavior. We believe they offer a less biased and more efficient and reproducible framework for determining milestone attainment. Ultimately, nephrology training programs must agree on a standardized set of curricular milestones and develop measurable, objective, validated milestone assessment tools. These must be transparent and submitted to peer review, as must ACGME analyses that use aggregated milestone achievement data. If useful data are to be produced regarding effective nephrology training tools, collaborative educational research investments (including financial investments) will be required.
ACKNOWLEDGEMENTS The views expressed in this report are those of the authors and do not reflect the official policy of the Department of the Army, the Department of the Navy, the Department of Defense, or the United States Government. Support: None. Financial Disclosure: The authors declare that they have no relevant financial interests.
SUPPLEMENTARY MATERIAL Figure S1: Sample target charts. Item S1: Walter Reed National Military Medical Center Nephrology Curricular Subcompetency Milestones (including MK1). Note: The supplementary material accompanying this article (http://dx.doi.org/10.1053/j.ajkd.2015.01.020) is available at www.ajkd.org
REFERENCES 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366:1051-1056. 2. The Internal Medicine Subspecialty Milestones Project: a joint initiative of the Accreditation Council for Graduate Medical Education and the American Board of Internal Medicine, in collaboration with the Alliance for Academic Internal Medicine. October 2014. http://acgme.org/acgmeweb/Portals/0/PDFs/Milestones/Internal MedicineSubspecialtyMilestones.pdf. Accessed December 16, 2014. 3. Vasilias J, Arrighi JA. Residency Review Committee for Internal Medicine (RRC-IM) update. APDIM fall meeting, October 2013. https://www.acgme.org/acgmeweb/Portals/0/ PFAssets/Presentations/140_APDIM_Oct2013_RCIM_Update.pdf. Accessed March 1, 2014. 4. Nephrology Curricular Milestones: implementation draft. American Society of Nephrology Training Program Directors. May 2014. https://www.asn-online.org/education/training/tpd/ ASN_Nephrology_Curricular_Milestones.pdf Accessed August 28, 2014. 5. American Society of Nephrology Training Program Directors Toolkit. https://www.asn-online.org/education/training/tpd/ toolkit.aspx. Accessed September 2, 2014.
22
6. Next Accreditation System: milestones. The Accreditation Council for Graduate Medical Education. http://www.acgme.org/ acgmeweb/tabid/430/ProgramandInstitutionalAccreditation/Next AccreditationSystem/Milestones.aspx. Accessed December 26, 2014. 7. Yuan CM, Nee R, Abbott KC, Oliver JD, III. Milestones for nephrology training programs: a modest proposal. Am J Kidney Dis. 2013;62(6):1034-1038. 8. Prince LK, Abbott KC, Green F, et al. Expanding the role of objectively structured clinical examinations in nephrology training. Am J Kidney Dis. 2014;63(6):906-912. 9. Yuan CM, Prince LK, Zwettler AJ, Nee R, Oliver JD III, Abbott KC. Assessing achievement in nephrology training: using clinic chart audits to quantitatively screen competency. Am J Kidney Dis. 2014;64(5):737-743. 10. Centers for Medicare and Medicaid Services (CMS). ESRD QIP summary: payment years 2012-2016. http://www.cms. gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/ ESRDQIP/Downloads/ESRD-QIP-Sumary-PY2012-16.pdf. Accessed September 2, 2014. 11. University of Washington Department of Laboratory Medicine. Online training and competency assessment for the clinical lab and point of care. http://www.medtraining.org. Accessed February 14, 2015. 12. Barsuk JH, Ahya SN, Cohen ER, McGaghie WC, Wayne DB. Mastery learning of temporary hemodialysis catheter insertion by nephrology fellows using simulation technology and deliberate practice. Am J Kidney Dis. 2009;54(1):70-76. 13. Keister DM, Larson D, Dostal J, Baglia J. The radar graph: the development of an educational tool to demonstrate resident competency. J Grad Med Educ. 2012;4(2):220-226. 14. Dreyfus H, Drefus SE. Mind over machine. In: Five Steps From Novice to Expert. New York, NY: Free Press; 1986:16-51. 15. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: developmental milestones for internal medicine residency training. J Grad Med Educ. 2009;1(1):5-20. 16. Aggarwal R, Darzi LA. Measurement of surgical performance for delivery of a competency-based training curriculum. In: Athanasoiu T, Debas H, Darzi A, eds. Key Topics in Surgical Research and Methodology. Berlin, Germany: Springer-Verlag; 2010:115-126. 17. Benner P. Using the Dreyfus model of skill acquisition to describe and interpret skill acquisition and clinical judgment in nursing practice and education. Bull Sci Technol Soc. 2004;24(3):188-199. 18. Pena A. The Dreyfus model of clinical problem-solving skills acquisition: a critical perspective. Med Educ Online. 2010;15:1-11. http://dx.doi.org/10.3402/meo.v15i0.4846. 19. Carraccio CL, Benson BJ, Nixon LJ, Derstine PL. From the educational bench to the clinical bedside: translating the Dreyfus developmental model to the learning of clinical skills. Acad Med. 2008;83(8):761-767. 20. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682. 21. Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ. Evaluating obstetrical residency programs using patient outcomes. JAMA. 2009;302(12):1277-1283. 22. Hoffman S, Podgurski A. Big bad data: law, public health, and biomedical databases. J Law Med Ethics. 2013;41(suppl 1):56-60. 23. Fan J, Han F, Liu H. Challenges of big data analysis. Natl Sci Rev. 2014;193:293-314.
Am J Kidney Dis. 2015;66(1):15-22