- Email: [email protected]
Milestones for Nephrology Training Programs: A Modest Proposal
Editorial Milestones for Nephrology Training Programs: A Modest Proposal
T
he Accreditation Council for Graduate Medical Education (ACGME) introduced the Next Accreditation System in order to improve future physician practice using the peer-review system, base training program accreditation decisions on trainee educational outcomes, and reduce administrative burdens associated with the present system.1 Specialty-specific milestones will be introduced to monitor the progress of trainees in achieving acceptable performance in the 6 competencies: Patient Care (PC), Medical Knowledge (MK), Professionalism (P), Interpersonal Communication Skills (IPCS), PracticeBased Learning and Improvement (PBLI), and Systems Based Practice (SBP). Each must be mastered to a level that allows the trainee to practice unsupervised. Milestones are goal behaviors that specialty and subspecialty trainees should achieve at defined times during their progression through training.1 Milestones standardize and clarify expectations for faculty and trainees. The Clinical Competency Committee, composed of key clinical faculty, will report on each trainee’s progress in achieving the milestones at least semiannually.2 They will select a description of trainee behavior using a 1-9 rating scale spanning “Critical deficiencies” to “Aspirational” behavior (Box 1). The Clinical Competency Committee also will formally decide whether the trainee is progressing satisfactorily toward achieving competence for unsupervised practice.3 Entrustable professional activities are complex task sets that define core specialty functions and, like milestones, permit standardized performance expectations.4,5 They describe what the trainee must do to satisfactorily provide unsupervised specialty care. Multiple milestones in multiple competencies can be mapped to a given entrustable professional activity, and the Clinical Competency Committee may use performance with respect to the latter to document milestone achievement. Schema of more than 140 milestones and 16 entrustable professional activities have been proposed
Originally published online August 22, 2013. Address correspondence to Christina M. Yuan, MD, Nephrology SVC, Department of Medicine, Walter Reed National Military Medical Center, 8901 Wisconsin Ave, Bethesda, MD 20889. E-mail: [email protected] Published by Elsevier Inc. on behalf of the National Kidney Foundation, Inc. This is a US Government Work. There are no restrictions on its use. 0272-6386/$0.00 http://dx.doi.org/10.1053/j.ajkd.2013.06.022 1034
for internal medicine.6,7 The ACGME has decided on 22 reporting milestones to be adopted in academic year 2013-2014.3 Subspecialty milestones (including nephrology) will be developed in 2013-2014 and introduced in 2014-2015.8 Formal efforts to adapt the internal medicine milestones to the subspecialties are just beginning.9,10 The ACGME-promulgated milestones for internal medicine have not yet been validated, and according to the ACGME, the “Ready for Unsupervised Practice” milestone is a target, but not yet a requirement, for graduation.3 Data provided by the training programs will be used for prospective validation, although the ACGME does not indicate what outcomes will be measured. American Board of Internal Medicine (ABIM) board examination performance may be used, as well as analysis of the patterns and rates of progression toward milestone achievement for all specialty residents.3 Our program, which is small (3 fellows per year) and military based, has incorporated milestones into competency assessment for 3 years. We use quantitative objective milestones when possible. Our milestones (Table 1) and the rationale for selecting them are as follows. First, milestones should be nephrology focused. As internal medicine specialty residency graduates, nephrology fellows have already been deemed “ready for unsupervised practice” in internal medicine. Therefore, nephrology milestones should be nephrology focused, not repetitions of milestones already achieved in internal medicine. As examples, they should include management of chronic kidney disease, end-stage renal disease (ESRD) preparation, dialysis therapy initiation and maintenance, and care of transplant donors and recipients. Second, milestones should be quantitative and objective, when possible, to avoid evaluator bias. Although milestone achievement can be assessed by use of entrustable professional activities, “satisfactory development” should be determined based on quantitative and objective assessment. This reduces the effect of rater bias by individual evaluators and the Clinical Competency Committee. Third, some milestones must be nephrology procedure focused. Definitive procedures include acute and chronic renal replacement therapy, continuous renal replacement therapy, kidney biopsy, temporary dialysis access placement, and microscopic urinalysis. Milestones must document and consider threshold numbers for competence in these procedures.11 Fourth, more is not necessarily better. A large number of detailed milestones are not necessarily Am J Kidney Dis. 2013;62(6):1034-1038
Editorial Box 1. Example Question and Rating Options From ACGME Report Worksheet for Patient Care Milestone 1 (PC1) Assessment in Internal Medicine Residents Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s). {Rating 1} Level 1: Critical Deficiencies Does not collect accurate historical data Does not use physical exam to confirm history Relies exclusively on documentation of others to generate own database or differential diagnosis Fails to recognize patient’s central clinical problems {Rating 2} {Rating 3} Level 2 Inconsistently able to acquire accurate historical information in an organized fashion Does not perform an appropriately thorough physical exam or misses key physical exam findings Does not seek or is overly reliant on secondary data Inconsistently recognizes patients’ central clinical problem or develops limited differential diagnoses {Rating 4} {Rating 5} Level 3 Consistently acquires accurate and relevant histories from patients Seeks and obtains data from secondary sources when needed Consistently performs accurate and appropriately thorough physical exams Uses collected data to define a patient’s central clinical problem(s) {Rating 6} {Rating 7} Level 4: Ready for unsupervised practice Acquires accurate histories from patients in an efficient, prioritized, and hypothesis-driven fashion Performs accurate physical exams that are targeted to the patient’s complaints Synthesizes data to generate a prioritized differential diagnosis and problem list Effectively uses history and physical exam skills to minimize the need for further diagnostic testing {Rating 8} {Rating 9} Level 5: Aspirational Obtains relevant historical subtleties, including sensitive information that informs the differential diagnosis Identifies subtle or unusual physical exam findings Efficiently uses all sources of secondary data to inform differential diagnosis Role models and teaches the effective use of history and physical exam skills to minimize the need for further diagnostic testing Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; exam, examination. Source: Accreditation Council for Graduate Medical Education Internal Medicine Milestone Project.3
superior to simpler representative ones. Quality literature recommends simplicity and parsimony in metrics, using aggregate metrics when possible.12,13 Am J Kidney Dis. 2013;62(6):1034-1038
Faculty time constraints and number of evaluators also are limiting. Smaller numbers of milestones facilitate validation and monitoring. Fifth, outcome measures must be defined to determine milestone validity. At present, the only outcomes available are the ABIM Nephrology Board Examination and the American Society of Nephrology In-Training Examination (ITE). Both focus on the Medical Knowledge and Patient Care competencies. Other potential measures include postgraduate evaluations by peers and patients and quality assurance data supplied to the Centers for Medicare & Medicaid Services (CMS). Sixth, tools to assess milestones should be defined. Our program uses a number of tools to assess competency and document milestone achievement, as described in the following: Outpatient clinic chart audits for general nephrology, maintenance dialysis, and transplantation. The audit tool (an Excel spreadsheet) has been shown to be associated with improvement in quality indicators (independent of reviewer).14 It allows assessment of trends over time, in tabular and graphical form, and permits quantifiable identification of deficiencies in care, timeliness, or documentation (competencies PC and IPCS); enables compliance with specified practice guidelines (MK and PBLI), and monitors military administrative actions that restrict assignments in active-duty patients (SBP). A median of 1,554 charts have been audited per year for the past 5 years by 7-9 faculty members. Outpatient clinic chart audits demonstrate progressive improvement and responsibility. Based on compliance with practice guidelines and reduction in deficiencies, audit is reduced from 100% to 25% of charts during the second year of fellowship (PC). Progressive improvement is demonstrated by achieving a ,5% deficiency rate per month by 6 months within a given training year and declining thereafter. Objective structured clinical examination (OSCE). First-year fellows must pass an OSCE demonstrating appropriate responses to and medical knowledge of 4 dialysis emergencies (PC). Second-year fellows must pass an OSCE using the College of American Pathologists urinalysis quality assurance survey (PC). Conference presentation and evaluation. Fellows present at approximately 6 conferences yearly, mentored by faculty. Attendees evaluate these conferences (IPCS, MK, and P) for presentation and content and indicate whether the information is likely to change practice. The average number of evaluations per fellow per year is 32 6 5 (SD). Procedure logs. Fellows must successfully perform 5 native kidney biopsies, 2 transplant kidney 1035
Yuan et al Table 1. Proposed Milestones for Nephrology Fellowship Training Competency
FY1 Milestones
FY2 Milestones
Patient Care (PC)
Progressive decline in chart audit deficiencies (defined as ,5% of encounters deficient at 6 mo and holding/declining thereafter) Mini-CEX RRT and kidney biopsy counseling: progressive improvement in meeting checklist requirements (1-9 scale) Dialysis emergencies OSCE: successful completion ($75% of initial management decisions correct)
Reduction in 100% outpatient chart audit (mo 1-6) to 50% (mo 7-9) to 25% (mo 10-12), based on continued deficiency reduction (defined as for FY1) Threshold number of procedures met (see text) and successfully completed Mini-CEX RRT and kidney biopsy counseling: must satisfactorily perform all components of each checklist (ie, achieve $6 on a 1-9 scale) Urinalysis OSCE: successful completion ($75% correct)
Medical Knowledge (MK)
Progressive improvement in evaluations of prepared academic lectures Progressive decline in chart audit deficiencies (see above)
Professionalism (P)
Progressive improvement to satisfactory performance on 360 evaluations
Lecture content and presentation at faculty level, prepared independently, during the last half of the year (average audience ratings $7; faculty input minimal; .50% of raters indicate that the lecture [when applicable] had the potential to change practice) Reduction in 100% outpatient chart audit (mo 1-6) to 50% (mo 7-9) to 25% (mo 10-12), based on continued deficiency reduction (see above) Preparation of independent research protocol; publication or presentation of abstract, case report, or journal article Successful completion of Medical Ethics curriculum (including $75% correct on military medical ethics test) Faculty-level performance on 360 evaluations by the second half of the year ($7 ratings in all areas)
Interpersonal and Communication Skills (IPCS)
Mini-CEX: scores progressively improve Chart audit: progressive improvement in timeliness and consultation skills
Mini-CEX: scores progressively improve and are fully acceptable by end of year ($6 on 1-9 scale) Chart audit: fully acceptable in timeliness and consultation skills (,5% deficiency; see above)
Practice-Based Learning and Improvement (PBLI)
Meets quality assurance metrics threshold on chart audit (eg, hypertension management, dialysis adequacy) Increasing percentage of “Yes” responses to the question “Will this presentation change your practice?” when applicable
Meets quality assurance metrics threshold on chart audit (eg, hypertension management, dialysis adequacy) .50% “Yes” responses to the question “Will this presentation change your practice?” when applicable Completion and presentation of nephrology-related, collaborative, multidisciplinary performance improvement project
Systems Based Practice (SBP)
Proper and timely completion of CMS Form 2728 Progressively improved administrative actions related to kidney disease
Correct completion of all administrative actions related to kidney disease
Abbreviations: CMS, Centers for Medicare & Medicaid Services; FY, fellowship year; Mini-CEX, Mini Clinical Evaluation Exercise; OSCE, Objective Structured Clinical Examination; RRT, renal replacement therapy.
biopsies, and 5 dialysis catheter placements and must have sufficient experience with continuous renal replacement therapy (PC). Thresholds were established in the early 1990s. Berns and O’Neill15 found that approximately one-third of nephrology training programs had not established minimum fellow procedure numbers. The majority of programs that had a threshold required 1-6 procedures per fellow.15 During the last 1036
5 years, the median number of procedures per fellow in our program was as follows: native kidney biopsies, 9 (range, 5-19); transplant kidney biopsies, 7 (range, 5-13); and temporary dialysis catheters, 8 (range, 5-18). Mini-Clinical Evaluation Exercise (Mini-CEX). Fellow performance in counseling patients and families before kidney biopsy and renal replacement therapy is Am J Kidney Dis. 2013;62(6):1034-1038
Editorial
assessed longitudinally using a faculty-administered Mini-CEX (IPCS, MK, PC, and P). Research productivity. Fellows must perform a mentored independent research project during their second year and present or publish a meeting abstract, case report, or journal article (MK, IPCS, and P). Multidisciplinary performance improvement project. Second-year fellows must perform and present results of a collaborative multidisciplinary nephrologyrelated performance improvement project (PBLI, P, and MK). Ethics curriculum. Second-year fellows must complete 3 ethics exercises: pass a military medical ethics review with a post-test, write a reflection on the documentary D Tour16 regarding maintenance dialysis and transplantation from the patient perspective, and respond to a clinical vignette of ESRD in an elderly patient, with a written description of case management and literature references (P). Completion of ESRD Medicare Evidence Form 2728. Accuracy and timeliness in completing ESRD Medicare Evidence Form 2728 are determined by social work services at end of first year (SBP). 360 evaluations by nursing, peer, and ancillary staff. Comprehensive evaluations are administered twice during each training year (P and IPCS). These milestones inform faculty and the Clinical Competency Committee whether trainees are advancing toward, and finally achieving, “competency for unsupervised practice.” They permit specific identification of deficiencies and demonstration of amendment and improvement. Some (Mini-CEX, conference presentation, and performance improvement project) are entrustable professional activities. Although not comprehensive, they are representative of and likely to predict the capacity for unsupervised nephrology practice. All proposed milestones should be prospectively validated with a priori outcome measures.17 Potential outcome measures include ITE and Nephrology Board Examination performance, end-of training procedure skills testing using simulation technology,11 postgraduate peer and patient evaluations, and performance data supplied to the CMS. The ITE, given twice during fellowship, could be used to demonstrate progressive improvement. For sufficient trainee numbers to permit comparisons, training program outcome results would have to be combined. Programs could be stratified, milestones under investigation could be randomized in blocks, agreed-upon outcomes could be measured, and evidence-based milestones could be adopted. Ideally, the structure and results of the educational research process would be transparent to all. Otherwise, we will be clicking radio buttons in rating software, unsure of whether the results predict whether our trainees are ready for unsupervised practice. It is imperative that nephrology Am J Kidney Dis. 2013;62(6):1034-1038
training programs have milestones of proven worth, not administrative millstones of dubious utility. Christina M. Yuan, MD Robert Nee, MD Kevin C. Abbott, MD, MPH James D. Oliver III, MD, PhD Nephrology Service Walter Reed National Military Medical Center Bethesda, Maryland
ACKNOWLEDGEMENTS The views expressed in this article are those of the authors and do not reflect the official policy of the Department of the Army, the Department of the Navy, the Department of Defense, or the United States government. Support: None. Financial Disclosure: The authors declare that they have no relevant financial interests.
REFERENCES 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366(11):1051-1056. 2. Accreditation Council for Graduate Medical Education. Frequently asked questions about the Next Accreditation System (updated December 2012). http://www.acgme-nas.org/assets/pdf/ NASFAQs.pdf. Accessed May 13, 2013. 3. Accreditation Council for Graduate Medical Education and American Board of Internal Medicine. The Internal Medicine Milestone Project (v. 12/2012). http://www.acgmenas.org/assets/pdf/Milestones/InternalMedicineMilestones.pdf. Accessed February 7, 2013. 4. ten Cate O. Entrustability of professional activities and competency-based training. Med Educ. 2005;39(12):1176-1177. 5. Carraccio C, Burke AE. Beyond competencies and milestones: adding meaning through context. J Grad Med Educ. 2010;2(3):419-422. 6. ABIM/ACGME Milestones Task Force. Developmental milestones for internal medicine residency training (draft), 2013. David Geffen School of Medicine at UCLA Graduate Medical Education. http://www.gme.medsch.ucla.edu/director/ documents/milestones-framework-draft.pdf. Accessed February 7, 2013. 7. Alliance for Academic Internal Medicine Education Redesign Committee. Internal medicine end-of-training EPAs, 2013. http://www.im.org/AcademicAffairs/milestones/Pages/ EndofTrainingEPAs.aspx. Accessed May 18, 2013. 8. Accreditation Council for Graduate Medical Education. RRC news: internal medicine. July 2012. http://www.acgme. org/acgmeweb/Portals/0/PFAssets/PublicationsNewsletters/ Internal_Medicine_Newsletter_Jul12_.pdf. Accessed February 7, 2013. 9. Alliance for Academic Internal Medicine. Internal medicine summit on subspecialty milestones (February 11-12, 2013, Alexandria, VA). http://www.im.org/AcademicAffairs/milestones/ Pages/imsubspecialtysummit.aspx. Accessed February 22, 2013. 10. American Society of Nephrology. The future of nephrology training: 2013 ASN training program directors’ retreat; Chicago, IL. http://www.asn-online.org/education/training/tpd/2013_TPD_ Retreat_Agenda.pdf. Accessed May 22, 2013. 1037
Yuan et al 11. Barsuk JH, Ahya SN, Cohen ER, McGaghie WC, Wayne DB. Mastery learning of temporary hemodialysis catheter insertion by nephrology fellows using simulation technology and deliberate practice. Am J Kidney Dis. 2009;54(1):70-76. 12. Brown MG. How to design your own measurement system. In: Keeping Score: Using the Right Metrics to Drive World-Class Performance. New York, NY: Productivity Press; 1996:3-16, 141-158. 13. Pronovost PJ, Bo-Linn GW. Preventing patient harms through systems of care. JAMA. 2012;308(8):769-770.
1038
14. Abbott KC, Bohen EM, Welch PG, Yuan CM. Analyzing process variation in chart review using a networked spreadsheet application in the Walter Reed Army Medical Center Nephrology Clinic. Mil Med. 2001;166(9):771-773. 15. Berns JS, O’Neill WC. Performance of procedures by nephrologists and nephrology fellows at U.S. nephrology training programs. Clin J Am Soc Nephrol. 2008;3(4):941-947. 16. Granato J. D Tour. San Francisco, CA: Autonomy16 Productions; 2009. 17. Parker MG. Nephrology training in the 21st century: toward outcomes-based education. Am J Kidney Dis. 2010;56(1):132-142.
Am J Kidney Dis. 2013;62(6):1034-1038
T
he Accreditation Council for Graduate Medical Education (ACGME) introduced the Next Accreditation System in order to improve future physician practice using the peer-review system, base training program accreditation decisions on trainee educational outcomes, and reduce administrative burdens associated with the present system.1 Specialty-specific milestones will be introduced to monitor the progress of trainees in achieving acceptable performance in the 6 competencies: Patient Care (PC), Medical Knowledge (MK), Professionalism (P), Interpersonal Communication Skills (IPCS), PracticeBased Learning and Improvement (PBLI), and Systems Based Practice (SBP). Each must be mastered to a level that allows the trainee to practice unsupervised. Milestones are goal behaviors that specialty and subspecialty trainees should achieve at defined times during their progression through training.1 Milestones standardize and clarify expectations for faculty and trainees. The Clinical Competency Committee, composed of key clinical faculty, will report on each trainee’s progress in achieving the milestones at least semiannually.2 They will select a description of trainee behavior using a 1-9 rating scale spanning “Critical deficiencies” to “Aspirational” behavior (Box 1). The Clinical Competency Committee also will formally decide whether the trainee is progressing satisfactorily toward achieving competence for unsupervised practice.3 Entrustable professional activities are complex task sets that define core specialty functions and, like milestones, permit standardized performance expectations.4,5 They describe what the trainee must do to satisfactorily provide unsupervised specialty care. Multiple milestones in multiple competencies can be mapped to a given entrustable professional activity, and the Clinical Competency Committee may use performance with respect to the latter to document milestone achievement. Schema of more than 140 milestones and 16 entrustable professional activities have been proposed
Originally published online August 22, 2013. Address correspondence to Christina M. Yuan, MD, Nephrology SVC, Department of Medicine, Walter Reed National Military Medical Center, 8901 Wisconsin Ave, Bethesda, MD 20889. E-mail: [email protected] Published by Elsevier Inc. on behalf of the National Kidney Foundation, Inc. This is a US Government Work. There are no restrictions on its use. 0272-6386/$0.00 http://dx.doi.org/10.1053/j.ajkd.2013.06.022 1034
for internal medicine.6,7 The ACGME has decided on 22 reporting milestones to be adopted in academic year 2013-2014.3 Subspecialty milestones (including nephrology) will be developed in 2013-2014 and introduced in 2014-2015.8 Formal efforts to adapt the internal medicine milestones to the subspecialties are just beginning.9,10 The ACGME-promulgated milestones for internal medicine have not yet been validated, and according to the ACGME, the “Ready for Unsupervised Practice” milestone is a target, but not yet a requirement, for graduation.3 Data provided by the training programs will be used for prospective validation, although the ACGME does not indicate what outcomes will be measured. American Board of Internal Medicine (ABIM) board examination performance may be used, as well as analysis of the patterns and rates of progression toward milestone achievement for all specialty residents.3 Our program, which is small (3 fellows per year) and military based, has incorporated milestones into competency assessment for 3 years. We use quantitative objective milestones when possible. Our milestones (Table 1) and the rationale for selecting them are as follows. First, milestones should be nephrology focused. As internal medicine specialty residency graduates, nephrology fellows have already been deemed “ready for unsupervised practice” in internal medicine. Therefore, nephrology milestones should be nephrology focused, not repetitions of milestones already achieved in internal medicine. As examples, they should include management of chronic kidney disease, end-stage renal disease (ESRD) preparation, dialysis therapy initiation and maintenance, and care of transplant donors and recipients. Second, milestones should be quantitative and objective, when possible, to avoid evaluator bias. Although milestone achievement can be assessed by use of entrustable professional activities, “satisfactory development” should be determined based on quantitative and objective assessment. This reduces the effect of rater bias by individual evaluators and the Clinical Competency Committee. Third, some milestones must be nephrology procedure focused. Definitive procedures include acute and chronic renal replacement therapy, continuous renal replacement therapy, kidney biopsy, temporary dialysis access placement, and microscopic urinalysis. Milestones must document and consider threshold numbers for competence in these procedures.11 Fourth, more is not necessarily better. A large number of detailed milestones are not necessarily Am J Kidney Dis. 2013;62(6):1034-1038
Editorial Box 1. Example Question and Rating Options From ACGME Report Worksheet for Patient Care Milestone 1 (PC1) Assessment in Internal Medicine Residents Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s). {Rating 1} Level 1: Critical Deficiencies Does not collect accurate historical data Does not use physical exam to confirm history Relies exclusively on documentation of others to generate own database or differential diagnosis Fails to recognize patient’s central clinical problems {Rating 2} {Rating 3} Level 2 Inconsistently able to acquire accurate historical information in an organized fashion Does not perform an appropriately thorough physical exam or misses key physical exam findings Does not seek or is overly reliant on secondary data Inconsistently recognizes patients’ central clinical problem or develops limited differential diagnoses {Rating 4} {Rating 5} Level 3 Consistently acquires accurate and relevant histories from patients Seeks and obtains data from secondary sources when needed Consistently performs accurate and appropriately thorough physical exams Uses collected data to define a patient’s central clinical problem(s) {Rating 6} {Rating 7} Level 4: Ready for unsupervised practice Acquires accurate histories from patients in an efficient, prioritized, and hypothesis-driven fashion Performs accurate physical exams that are targeted to the patient’s complaints Synthesizes data to generate a prioritized differential diagnosis and problem list Effectively uses history and physical exam skills to minimize the need for further diagnostic testing {Rating 8} {Rating 9} Level 5: Aspirational Obtains relevant historical subtleties, including sensitive information that informs the differential diagnosis Identifies subtle or unusual physical exam findings Efficiently uses all sources of secondary data to inform differential diagnosis Role models and teaches the effective use of history and physical exam skills to minimize the need for further diagnostic testing Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; exam, examination. Source: Accreditation Council for Graduate Medical Education Internal Medicine Milestone Project.3
superior to simpler representative ones. Quality literature recommends simplicity and parsimony in metrics, using aggregate metrics when possible.12,13 Am J Kidney Dis. 2013;62(6):1034-1038
Faculty time constraints and number of evaluators also are limiting. Smaller numbers of milestones facilitate validation and monitoring. Fifth, outcome measures must be defined to determine milestone validity. At present, the only outcomes available are the ABIM Nephrology Board Examination and the American Society of Nephrology In-Training Examination (ITE). Both focus on the Medical Knowledge and Patient Care competencies. Other potential measures include postgraduate evaluations by peers and patients and quality assurance data supplied to the Centers for Medicare & Medicaid Services (CMS). Sixth, tools to assess milestones should be defined. Our program uses a number of tools to assess competency and document milestone achievement, as described in the following: Outpatient clinic chart audits for general nephrology, maintenance dialysis, and transplantation. The audit tool (an Excel spreadsheet) has been shown to be associated with improvement in quality indicators (independent of reviewer).14 It allows assessment of trends over time, in tabular and graphical form, and permits quantifiable identification of deficiencies in care, timeliness, or documentation (competencies PC and IPCS); enables compliance with specified practice guidelines (MK and PBLI), and monitors military administrative actions that restrict assignments in active-duty patients (SBP). A median of 1,554 charts have been audited per year for the past 5 years by 7-9 faculty members. Outpatient clinic chart audits demonstrate progressive improvement and responsibility. Based on compliance with practice guidelines and reduction in deficiencies, audit is reduced from 100% to 25% of charts during the second year of fellowship (PC). Progressive improvement is demonstrated by achieving a ,5% deficiency rate per month by 6 months within a given training year and declining thereafter. Objective structured clinical examination (OSCE). First-year fellows must pass an OSCE demonstrating appropriate responses to and medical knowledge of 4 dialysis emergencies (PC). Second-year fellows must pass an OSCE using the College of American Pathologists urinalysis quality assurance survey (PC). Conference presentation and evaluation. Fellows present at approximately 6 conferences yearly, mentored by faculty. Attendees evaluate these conferences (IPCS, MK, and P) for presentation and content and indicate whether the information is likely to change practice. The average number of evaluations per fellow per year is 32 6 5 (SD). Procedure logs. Fellows must successfully perform 5 native kidney biopsies, 2 transplant kidney 1035
Yuan et al Table 1. Proposed Milestones for Nephrology Fellowship Training Competency
FY1 Milestones
FY2 Milestones
Patient Care (PC)
Progressive decline in chart audit deficiencies (defined as ,5% of encounters deficient at 6 mo and holding/declining thereafter) Mini-CEX RRT and kidney biopsy counseling: progressive improvement in meeting checklist requirements (1-9 scale) Dialysis emergencies OSCE: successful completion ($75% of initial management decisions correct)
Reduction in 100% outpatient chart audit (mo 1-6) to 50% (mo 7-9) to 25% (mo 10-12), based on continued deficiency reduction (defined as for FY1) Threshold number of procedures met (see text) and successfully completed Mini-CEX RRT and kidney biopsy counseling: must satisfactorily perform all components of each checklist (ie, achieve $6 on a 1-9 scale) Urinalysis OSCE: successful completion ($75% correct)
Medical Knowledge (MK)
Progressive improvement in evaluations of prepared academic lectures Progressive decline in chart audit deficiencies (see above)
Professionalism (P)
Progressive improvement to satisfactory performance on 360 evaluations
Lecture content and presentation at faculty level, prepared independently, during the last half of the year (average audience ratings $7; faculty input minimal; .50% of raters indicate that the lecture [when applicable] had the potential to change practice) Reduction in 100% outpatient chart audit (mo 1-6) to 50% (mo 7-9) to 25% (mo 10-12), based on continued deficiency reduction (see above) Preparation of independent research protocol; publication or presentation of abstract, case report, or journal article Successful completion of Medical Ethics curriculum (including $75% correct on military medical ethics test) Faculty-level performance on 360 evaluations by the second half of the year ($7 ratings in all areas)
Interpersonal and Communication Skills (IPCS)
Mini-CEX: scores progressively improve Chart audit: progressive improvement in timeliness and consultation skills
Mini-CEX: scores progressively improve and are fully acceptable by end of year ($6 on 1-9 scale) Chart audit: fully acceptable in timeliness and consultation skills (,5% deficiency; see above)
Practice-Based Learning and Improvement (PBLI)
Meets quality assurance metrics threshold on chart audit (eg, hypertension management, dialysis adequacy) Increasing percentage of “Yes” responses to the question “Will this presentation change your practice?” when applicable
Meets quality assurance metrics threshold on chart audit (eg, hypertension management, dialysis adequacy) .50% “Yes” responses to the question “Will this presentation change your practice?” when applicable Completion and presentation of nephrology-related, collaborative, multidisciplinary performance improvement project
Systems Based Practice (SBP)
Proper and timely completion of CMS Form 2728 Progressively improved administrative actions related to kidney disease
Correct completion of all administrative actions related to kidney disease
Abbreviations: CMS, Centers for Medicare & Medicaid Services; FY, fellowship year; Mini-CEX, Mini Clinical Evaluation Exercise; OSCE, Objective Structured Clinical Examination; RRT, renal replacement therapy.
biopsies, and 5 dialysis catheter placements and must have sufficient experience with continuous renal replacement therapy (PC). Thresholds were established in the early 1990s. Berns and O’Neill15 found that approximately one-third of nephrology training programs had not established minimum fellow procedure numbers. The majority of programs that had a threshold required 1-6 procedures per fellow.15 During the last 1036
5 years, the median number of procedures per fellow in our program was as follows: native kidney biopsies, 9 (range, 5-19); transplant kidney biopsies, 7 (range, 5-13); and temporary dialysis catheters, 8 (range, 5-18). Mini-Clinical Evaluation Exercise (Mini-CEX). Fellow performance in counseling patients and families before kidney biopsy and renal replacement therapy is Am J Kidney Dis. 2013;62(6):1034-1038
Editorial
assessed longitudinally using a faculty-administered Mini-CEX (IPCS, MK, PC, and P). Research productivity. Fellows must perform a mentored independent research project during their second year and present or publish a meeting abstract, case report, or journal article (MK, IPCS, and P). Multidisciplinary performance improvement project. Second-year fellows must perform and present results of a collaborative multidisciplinary nephrologyrelated performance improvement project (PBLI, P, and MK). Ethics curriculum. Second-year fellows must complete 3 ethics exercises: pass a military medical ethics review with a post-test, write a reflection on the documentary D Tour16 regarding maintenance dialysis and transplantation from the patient perspective, and respond to a clinical vignette of ESRD in an elderly patient, with a written description of case management and literature references (P). Completion of ESRD Medicare Evidence Form 2728. Accuracy and timeliness in completing ESRD Medicare Evidence Form 2728 are determined by social work services at end of first year (SBP). 360 evaluations by nursing, peer, and ancillary staff. Comprehensive evaluations are administered twice during each training year (P and IPCS). These milestones inform faculty and the Clinical Competency Committee whether trainees are advancing toward, and finally achieving, “competency for unsupervised practice.” They permit specific identification of deficiencies and demonstration of amendment and improvement. Some (Mini-CEX, conference presentation, and performance improvement project) are entrustable professional activities. Although not comprehensive, they are representative of and likely to predict the capacity for unsupervised nephrology practice. All proposed milestones should be prospectively validated with a priori outcome measures.17 Potential outcome measures include ITE and Nephrology Board Examination performance, end-of training procedure skills testing using simulation technology,11 postgraduate peer and patient evaluations, and performance data supplied to the CMS. The ITE, given twice during fellowship, could be used to demonstrate progressive improvement. For sufficient trainee numbers to permit comparisons, training program outcome results would have to be combined. Programs could be stratified, milestones under investigation could be randomized in blocks, agreed-upon outcomes could be measured, and evidence-based milestones could be adopted. Ideally, the structure and results of the educational research process would be transparent to all. Otherwise, we will be clicking radio buttons in rating software, unsure of whether the results predict whether our trainees are ready for unsupervised practice. It is imperative that nephrology Am J Kidney Dis. 2013;62(6):1034-1038
training programs have milestones of proven worth, not administrative millstones of dubious utility. Christina M. Yuan, MD Robert Nee, MD Kevin C. Abbott, MD, MPH James D. Oliver III, MD, PhD Nephrology Service Walter Reed National Military Medical Center Bethesda, Maryland
ACKNOWLEDGEMENTS The views expressed in this article are those of the authors and do not reflect the official policy of the Department of the Army, the Department of the Navy, the Department of Defense, or the United States government. Support: None. Financial Disclosure: The authors declare that they have no relevant financial interests.
REFERENCES 1. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366(11):1051-1056. 2. Accreditation Council for Graduate Medical Education. Frequently asked questions about the Next Accreditation System (updated December 2012). http://www.acgme-nas.org/assets/pdf/ NASFAQs.pdf. Accessed May 13, 2013. 3. Accreditation Council for Graduate Medical Education and American Board of Internal Medicine. The Internal Medicine Milestone Project (v. 12/2012). http://www.acgmenas.org/assets/pdf/Milestones/InternalMedicineMilestones.pdf. Accessed February 7, 2013. 4. ten Cate O. Entrustability of professional activities and competency-based training. Med Educ. 2005;39(12):1176-1177. 5. Carraccio C, Burke AE. Beyond competencies and milestones: adding meaning through context. J Grad Med Educ. 2010;2(3):419-422. 6. ABIM/ACGME Milestones Task Force. Developmental milestones for internal medicine residency training (draft), 2013. David Geffen School of Medicine at UCLA Graduate Medical Education. http://www.gme.medsch.ucla.edu/director/ documents/milestones-framework-draft.pdf. Accessed February 7, 2013. 7. Alliance for Academic Internal Medicine Education Redesign Committee. Internal medicine end-of-training EPAs, 2013. http://www.im.org/AcademicAffairs/milestones/Pages/ EndofTrainingEPAs.aspx. Accessed May 18, 2013. 8. Accreditation Council for Graduate Medical Education. RRC news: internal medicine. July 2012. http://www.acgme. org/acgmeweb/Portals/0/PFAssets/PublicationsNewsletters/ Internal_Medicine_Newsletter_Jul12_.pdf. Accessed February 7, 2013. 9. Alliance for Academic Internal Medicine. Internal medicine summit on subspecialty milestones (February 11-12, 2013, Alexandria, VA). http://www.im.org/AcademicAffairs/milestones/ Pages/imsubspecialtysummit.aspx. Accessed February 22, 2013. 10. American Society of Nephrology. The future of nephrology training: 2013 ASN training program directors’ retreat; Chicago, IL. http://www.asn-online.org/education/training/tpd/2013_TPD_ Retreat_Agenda.pdf. Accessed May 22, 2013. 1037
Yuan et al 11. Barsuk JH, Ahya SN, Cohen ER, McGaghie WC, Wayne DB. Mastery learning of temporary hemodialysis catheter insertion by nephrology fellows using simulation technology and deliberate practice. Am J Kidney Dis. 2009;54(1):70-76. 12. Brown MG. How to design your own measurement system. In: Keeping Score: Using the Right Metrics to Drive World-Class Performance. New York, NY: Productivity Press; 1996:3-16, 141-158. 13. Pronovost PJ, Bo-Linn GW. Preventing patient harms through systems of care. JAMA. 2012;308(8):769-770.
1038
14. Abbott KC, Bohen EM, Welch PG, Yuan CM. Analyzing process variation in chart review using a networked spreadsheet application in the Walter Reed Army Medical Center Nephrology Clinic. Mil Med. 2001;166(9):771-773. 15. Berns JS, O’Neill WC. Performance of procedures by nephrologists and nephrology fellows at U.S. nephrology training programs. Clin J Am Soc Nephrol. 2008;3(4):941-947. 16. Granato J. D Tour. San Francisco, CA: Autonomy16 Productions; 2009. 17. Parker MG. Nephrology training in the 21st century: toward outcomes-based education. Am J Kidney Dis. 2010;56(1):132-142.
Am J Kidney Dis. 2013;62(6):1034-1038