Novice to Expert: The Evolution of an Advanced Practice Evaluation Tool

ARTICLE

Novice to Expert: The Evolution of an Advanced Practice Evaluation Tool Juanita Conkin Dale, PhD, RN, CPNP-PC, Barbie Drews, MS, RN, CPNP-PC, Paula Dimmitt, MS, RN, CPNP-PC, Evelyn Hildebrandt, MS, RN, PNP-BC, Kristin Hittle, MS, RN, CPNP-AC, & Anna Tielsch-Goddard, MSN, RN, CPNP-PC ABSTRACT

KEY WORDS

Professional performance evaluation provides an opportunity to measure the practice of health providers within healthcare settings. Standardized evaluation can be challenging as a result of diverse practice arenas, multiple evaluators and standards of care. Using BennerÕs novice to expert model, a Performance Excellence and Accountability tool (PEAC Toolq) has been designed to measure advanced practice providers performance based upon facets of professional practice. This article discusses development, practical implementation and evaluation of a PEAC Toolq. J Pediatr Health Care. (2013) 27, 195-201.

Advanced practice nurse, physician assistant, performance evaluation tools, SMART goals

Juanita Conkin Dale, Pediatric Nurse Practitioner, Comprehensive Epilepsy Program, ChildrenÕs Medical Center, Dallas, TX. Barbie Drews, Pediatric Nurse Practitioner, Center for Pediatric Gastroenterology and Nutrition, ChildrenÕs Medical Center, Dallas, TX. Paula Dimmitt, CPNP-PC, Manager, Advanced Practice Service, ChildrenÕs Medical Center, Dallas, TX. Evelyn Hildebrandt, Pediatric Nurse Practitioner, Comprehensive Pediatric Epilepsy Program, ChildrenÕs Medical Center, Dallas, TX. Kristin Hittle, Pediatric Nurse Practitioner, Pediatric Intensive Care Unit, ChildrenÕs Medical Center, Dallas, TX. Anna Tielsch-Goddard, Perioperative Services, ChildrenÕs Medical Center at Legacy, Plano, TX. Conflicts of interest: None to report. Correspondence: Juanita Conkin Dale, PhD, RN, CPNP-PC, 1935 Medical District Drive, Dallas, TX 75235; e-mail: juanita.dale@ childrens.com. 0891-5245/$36.00 Copyright Q 2013 by the National Association of Pediatric Nurse Practitioners. Published by Elsevier Inc. All rights reserved. Published online January 9, 2012. http://dx.doi.org/10.1016/j.pedhc.2011.12.004

www.jpedhc.org

The numbers of nurse practitioners (NPs), clinical nurse specialists (CNSs), and physician assistants (PAs), collectively known as advanced practice professionals (APPs), have grown in childrenÕs hospitals. This growth was primarily stimulated by the Accreditation Council for Graduate Medical Education (ACGME) work hour restrictions imposed for resident physicians in 2001. The ACGME Duty Hours Standards (2011) were recently revised further limiting the time that residents can be scheduled for continuous duty in the hospital thus increasing the need for other providers in childrenÕs hospitals. In 2005, the Advanced Practice Service Department (APS) at ChildrenÕs Medical Center (CMC) was established to create an organized, coordinated approach to manage the number of APPs that grew from 40 in 2001 to 90 in 2005 (and currently over 200). The managerial oversight of APPs is unique in that the role includes the medical management of patients. Therefore the objectives of the APS were to effectively evaluate the APPÕs medical management skills and knowledge, create a professional development model with defined expectations, comply with the Joint Commission credentialing requirements, and to devise a process of billing for professional services. APS employees work in outpatient and inpatient settings that include, but are not limited to, intensive care units and specialty care services, emergency departments, ambulatory care clinics, and the perioperative service departments. Although PAs, NPs, and CNSs are uniquely educated, licensed, regulated, and supervised, all three disciplines are required to maintain the May/June 2013

195

same professional standards of employment as an APP at CMC. The shared governance vehicle for APS is the Advanced Practice Advisory Group (APAG). The APAG is a representative group of APPs from various services throughout the medical center that was formed to support the discussions and development of polices as they relate to advanced practice. One early challenge for APAG was to create an objective, equitable process for evaluating both the clinical performance of an APP as well as the expected professional activities. Therefore, a subgroup of APAG members developed a professional evaluation tool to complement the hospital annual performance guide. The Benner (2001) Model was identified as a theoretical framework to outline advancement based on clinical practice and expertise of APPs from ‘‘Novice to Expert.’’ REVIEW OF LITERATURE A Cochrane review was utilized for search of key terms including ‘‘SMART goals’’ (SMART standing for: Specific, Measurable, Attainable, Relevant, and Timebound); ‘‘evaluations’’ using NP, PAs, nurses, health care clinicians, and physicians as precedents; ‘‘peer review’’; ‘‘employee performance’’; ‘‘performance review’’; ‘‘performance appraisal’’; and ‘‘constructive feedback.’’ A secondary source search was utilized from initial articles that were found. This process was repeated in a NCBI PubMed review, as well as a Questia search to include references from economic and business resources. A review of the literature revealed no specific evaluation process that objectively measured professional and clinical development based on the multi-faceted role of an APP. There is agreement that among advanced practice nurses there are four professional roles: NP, CNS, nurse anesthetist, and nurse midwife (American Association of Colleges of Nursing, 2011). However, the roles that these APNs play extend beyond the clinical care they provide and a method for evaluating their non-clinical role is not published. Professions such as teachers, speech therapists, and psychologists have objective forms for evaluation, which are widely published. Several tools are used nation-wide for evaluation of the registered nurse (RN). However, none of these focused on the advanced practice role dimensions and growth of an APP. Evaluation Tools Crumbie and Kyle (2006) found that evaluations are often looked at as a valuable tool to allow the employer the opportunity to assess employee contributions to the organization. They also allow employees to share their goals for individual professional development over the next year and should become an important aspect of a higher-quality practice. 196

Volume 27  Number 3

When considering employee evaluation approaches, research emphasizes the importance of objectivity throughout the process (Crumbie & Kyle, 2006; Black, 2001; Capko, 2003). The process should begin with a written job description and include a point-based rating system reflective of the requirements of that specific job (Black, 2001; Capko, 2003). The appraiser must have an objective measurable descriptive range; i.e., meets basic expectations, sometimes exceeds expectations, and always exceeds expectations (Capko, 2003). Black (2001) recommends identifying goals for the next appraisal period that are specific and measurable. Goal setting is beneficial for the employeeÕs career path and often improves retention (Black, 2001). The SMART goal is often used to help employees devise a set goal and fulfill specific criteria. These guidelines for goal development give employees opportunities to create their own objectives, criteria, and a timescale for their goals (Pearce, 2007). In developing an evaluation form, focus should be on the essential job performance areas while assessing the most meaningful and relevant issues. According to Capko (2003) the performance evaluation form should measure five elements of job performance including job knowledge and skills, quality of work, quantity of work, attitude, and work habits. Guidelines set forth by Crumbie and Kyle (2006) emphasize incorporating many facets of NP functions within the evaluation. They listed ‘‘feedback from colleagues and patients, significant events, audits, study days/conferences, courses, teaching, protocols developed, presentations, and publications’’ (Crumbie & Kyle, 2006, p. 15) as essential components to the NP evaluation. Fletcher (2004) suggests an effective appraisal tool includes evaluating performance, how well the work environment supports staff development, a reflective process to look at the employeeÕs growth, personal learning needs, active participation, constructive feedback, and self-assessment. Evaluations provide measurement and documentation of the employeeÕs contribution to the team and overall organization. This protects both the employee and employer in providing appraisal documentation while encouraging a higher level of quality and quantity of work produced (Capko, 2003). Employees learn to take pride in their work and will accept new challenges with greater confidence after positive feedback is received (Capko, 2003). Morgan Roberts and colleagues (2005) point out that while people often focus on criticism, they gain more confidence from praise. Managers should use the employeeÕs strengths to contribute more to the organization and identify opportunities for growth (Morgan Roberts et al., 2005). For a successful review of an APP, the appraiser needs to understand the unique nature of the APPÕs role, which often presents a challenge (Crumbie & Kyle, 2006). The importance of a health care team Journal of Pediatric Health Care

TABLE 1. Benner Model of professional nursing skills Level of experience

Who they are traditionally?

Novice

New graduate or new provider to a specialty

Advance Beginner Competent

2–3 years of experience

Proficient Expert

Many years of experience

How they practice Governed by rules for practice and lacks the experience to modify them as needed Demonstrates acceptable care performance but has difficulty transferring knowledge from a previous experience to a current practice, largely rules-driven Prioritizes and identifies aspects of the care easily incorporating previous experiences in decision-making Able to prioritize care almost automatically and recognizes deviations from a norm Possess an intuitive grasp of a situation based upon experience and knowledge gained over time

Source: Benner P. (2001). From Novice to Expert: Excellence and power in clinical nursing practice. Upper Saddle River, New Jersey: Prentice Hall Health.

developing their own evaluation strategy to reflect practice philosophy is vital to the continued success of the appraisal process (Fletcher, 2004). Benner Model Dreyfus and Dreyfus (1980) studied chess players and airline pilots to define the process of attaining expertise in a field. They described characteristics at each level of proficiency from beginning to master. Benner (2001) modified the Dreyfus Model of Skill Acquisition in developing her model to measure professional nursing skill acquisition for a more objective way of evaluating progress in learning any new subject or skill. The Benner Model contains five levels to define stages of growth (see Table 1) that can be applied to any health care professional role. In the Benner Model, the novice is a new RN whose practice is rule-driven. The advanced beginner demonstrates acceptable performance, but lacks a well-developed knowledge base and strong management skills. The competent professional can prioritize and use past knowledge to care for patients. The proficient professional will evaluate the patient situation, then automatically prioritize needs and predict outcomes. The expert professional exhibits confidence and has developed an extensive knowledge base allowing for an intuitive grasp of complex patient situations. Importance of Peer- and Self-Review APPs should have representation by peers of similar specialty and rank as part of the peer-evaluation process (Gentry, 2006; Kenny, Baker, Lanzon, Stevens, & Yancy, 2008). Peer evaluation, as a component of nurse job performance, was first supported as a necessity by the National League for Nursing (1979). The American Nurses Association also supports nurses of similar rank and of clinical expertise conducting peer evaluations (Kenny et al., 2008). In some health care settings, APPs may not have a nursing or PA colleague to participate in their appraisal, making peer evaluations more difficult (Fletcher, 2004). Although the manager should offer insight into the clinicianÕs www.jpedhc.org

progress, the employee should also be able to address the positive and negative aspects of the work experience including a thoughtful self-evaluation (Fletcher, 2004). Appraisals that focus on the sole input of a higher ranked employee can be viewed as biased, subjective, and unfair (King, 2002). Peer review has been found to increase professionalism, enhance teamwork, and contribute to a non-competitive work environment (Gentry, 2006). Performance Excellence and Accountability (PEAC) Toolq Clinical advancement and annual evaluations in nursing have traditionally been subjective, not based on objective measurable outcomes. The evaluations were often performed by managers who were not advanced practice nurses and therefore not equipped to identify specific contributions of an effective APP. The APS adapted a PEAC Toolq to objectively measure APP performance. The APAG charged a task force comprised of APPs and an APS manager to modify the current PEAC Toolq. The pre-existing performance objectives and job descriptions were reviewed. The task force solicited input from all ambulatory and inpatient APPs and the APS management team. Over 18 months, an evaluation tool was developed to incorporate the different facets of advanced practice. APPs are evaluated with the PEAC Toolq. In the APP job description, the duties and tasks are identified within the following roles: practitioner, consultant, educator, researcher, and professional accountability (see Box 1). These roles were incorporated into the PEAC Toolq. Within each role, there are up to four potential behaviors. Table 2 exemplifies one expected behavior for each role in the PEAC Toolq. Target Performance for meeting SMART goals was rated on a Behaviorally Anchored Rating Scale (BARS) of ‘‘Did Not Meet Basic Expectations,’’ ‘‘Met Basic Expectations,’’ ‘‘Met/Sometimes Exceeded Expectations,’’ ‘‘Consistently Exceeded Expectations,’’ and ‘‘Outstanding’’ to correspond with the progression of proficiency from novice to expert. May/June 2013

197

BOX 1. Duties and tasks of an advanced practice professional Practitioner: Provides medical and advanced care as guided by education, clinical expertise and collaborative agreement  Performs a complete or an interim physical examination as indicated by the nature of the patient encounter.  Diagnoses acute and chronic health problems.  Orders necessary diagnostic and/or therapeutic tests (laboratory, radiological, or other).  Makes referrals as indicated to appropriate health care professionals.  Plans interventions including medications, treatments, and procedures.  Evaluates patient response to interventions and modifies care plan as necessary to achieve desired outcomes. Consultant: Consults as needed with other health care professionals, patients, and families in the formulation and initiation of the health care plan.  Evaluates the outcome of the patient consultation and modifies recommendations as needed to facilitate restoration of health.  Provides consultative resources to direct care nurses with the goal of enhancing knowledge and improving patient care outcomes. Educator:  Identifies the learning needs of patients and their families and directs educational activities to meet these needs.  Recognizes educational or knowledge needs of peers or nursing staff and participates in formal or informal educational activities to meet identified needs.  Evaluates response to educational interventions, including validation of learning and understanding to determine the impact on quality of care. Researcher:  Participates actively in research, independently or in collaboration with other professionals as a member of a research team, and/or by the utilization of research relevant to practice area.  Reviews and utilizes current literature and research findings for evidence-based practice in the health care of pediatric patients and their families. Professional Accountability:  Maintains all licensure and certifications required for current position.  Completes mandatory education and competencies.  Exhibits ongoing professional development: sustains organization memberships, attends educational offering, maintains an ethical practice.

The PEAC Toolq was adapted using BennerÕs five levels of proficiency. At the novice or advanced beginner level, the APP consistently meets basic expectations for the position. The competent level performance reflects algorithmic decision making for patient care and professional outcomes. At the proficient level, the APPÕs performance reflects integration of experiences in decision making for patient care and professional outcomes. For an APP practicing at the expert level, performance reflects intuitive thought processes in decision-making for patient care and professional outcomes. APPs were then to be compensated based upon their respective levels of proficiency. After initial presentation of the PEAC Toolq, novice APPs expressed concerns that only APPs exhibiting expert level behaviors would receive higher pay increases. In addition, some of the APPs who were expert clinicians in their previous role, either RN or APP, and now performing at the novice level in a new role were overwhelmed at the targeted performances outlined in the PEAC Toolq. In keeping with the Benner Model, achieving competent status could take two to three years and achieving expert status would take longer. Novice APPs joined the task force to provide input for further refinement of the tool. Coinciding with the development of this PEAC Toolq was the 198

Volume 27  Number 3

creation of an APS Orientation Manual. Task force discussions included developing a separate tool for the novice APP with modified goals. With further consideration, it was decided that the novice APP would only be responsible for mastering the behaviors outlined in the APS Orientation Manual that focuses more on clinical and professional accountability. A novice APPÕs evaluation for the first year of service focuses on the attainment of expected competencies. For subsequent annual reviews, the PEAC Toolq is utilized as the standard. Ongoing revisions were made throughout the process and the tool was to be used by all APS managers for the 2009 evaluation period. Employee Feedback of the PEAC Toolq In order to obtain feedback of the PEAC Toolq, two separate surveys were developed; one for the APP staff and another for APS Management Team using SurveyMonkeyÒ. The main goal of the surveys was to determine if staff perceived the tool as objective and to identify how each manager was utilizing the tool to evaluate staff. The survey was confidential to aid in eliciting honest responses. Only APPs (n = 130) who had been evaluated using the PEAC Toolq were eligible. Of the eligible APPs, 99 (76%) responded to the survey. The APPs surveyed had practiced from 1 to 37 years. Journal of Pediatric Health Care

www.jpedhc.org

TABLE 2. Examples from the PEAC Toolq Levels of performance (Benner)

Novice or advanced beginner

Target Performance Distribution ChildrenÕs Medical Center PEAC Definitions

Did Not Meet Expectations

Met Basic Expectations

Work performance consistently below expectations for the position

Work performance consistently met basic expectations for the positions

Job Performance

Practitioner Significant elements of job description not being performed to satisfaction

Competent

Proficient

Expert

Met/Sometimes Exceeded Basic Expectations Work performance consistently met and sometimes exceeded expectations for the positions

Consistently Exceeded Expectations Work performance consistently exceeded expectations for the positions

Outstanding

Meets/performs standard elements of job description

Performance reflects algorithmic decisionmaking for patient care and professional outcomes

Performance reflects integration of experiences in decision-making for patient care and professional outcomes

Performance reflects intuitive thought processes in decision-making for patient care and professional outcomes

Applies learning consistently

Shares and applies learning by formal teaching/lecturing

Independent pursuit of personal growth by attendance at school, seminar(s), conference(s), or online continuing education

Mentoring NP or PA: Role development, publication, or research

Assists in identifying or developing nursing inservice

Provides graduate-level lecture at a college or university

Submits manuscript for publication or abstract for podium or poster presentation

Local or national conference lecture or poster presentation

References and applies evidence-based practice to care of patients

Researches evidence-based practice problem and applies solution

Co-investigator of a research study

Develops, writes a research proposal, and submits for IRB approval

Work performance consistently outstanding

Consultant

Educator

Researcher Clinical practice is not current with standard of care

May/June 2013

PEAC, Performance Excellence and Accountability; NP, nurse practitioner; PA, physicianÕs assistant; IRB, Institutional Review Board.

199

TABLE 3. Results from PEAC Toolq Evaluation Survey of advanced practice provider staff Employee perception of evaluation with PEAC Toolq PEAC Toolq was.

Strongly agree

Agree

Disagree

Strongly disagree

Easy to use Objective Comprehensive Able to measure professional growth Reflective of the four facets of advanced practice provider practice

9% (n = 8) 9% (n = 8) 9% (n = 8) 10% (n = 9) 7% (n = 6)

75% (n = 70) 74% (n = 69) 79% (n = 74) 61% (n = 57) 77% (n = 72)

15% (n = 14) 16% (n = 15) 11% (n = 10) 27% (n = 25) 14% (n = 13)

1% (n = 1) 1% (n = 1) 1% (n = 1) 2% (n = 2) 2% (n = 2)

PEAC, Performance Excellence and Accountability.

The APP follow-up survey included five items measured on a Likert scale with comments requested. The items surveyed were ‘‘easy to use,’’ ‘‘objective,’’ ‘‘comprehensive,’’ ‘‘ability to measure professional growth,’’ and ‘‘reflective of the four facets of practice (Practitioner, Consultant, Educator, Researcher).’’ Eighty-six percent of the respondents thought the PEAC Toolq was easy to use, objective, comprehensive, and reflected the four facets of practice (see Table 3). The most significant negative feedback on tool use was that 29% of employees believed that the PEAC Toolq was not effective for measuring professional growth. Sample comments from the staff survey are presented in Box 2. A persistent theme throughout the survey was that APPs wanted more emphasis on their clinical roles and not to be measured on their attendance at meetings, participation on committees, or as educators and researchers. One hundred percent of the managers (n = 6) believed that the PEAC Toolq was objective, applicable to service area, and made evaluation of staff easier. One of the six respondents did not believe that the tool was easy to use. Comments from the APS managers and recommendations for improving the PEAC Toolq are presented in Box 2. Fifty percent of the managers thought the tool should be used as a clinical ladder tool or professional development tool.

DISCUSSION The APS developed a useful tool to objectively evaluate NP, PA, and CNS (APPs) practice using a BARS scale. This PEAC form is now an accepted tool implemented to assist in scoring the APPÕs annual performance evaluation. It has also provided concrete examples of actions denoting novice to expert type practices to help guide and motivate progression in the APPÕs career. Three years of refining the tool with informal input from APPs and their managers has increased the objective measurement of the PEAC Toolq. The recent anonymous survey indicated the PEAC Toolq is objective. However, the perspective of how it should be used was inconsistent. Although most managers and APPs thought the tool was helpful and easy to use during the evaluation process, it did not adequately weigh the many clinical behaviors of the APP. The survey results indicated there was a ‘‘disconnect’’ with the measurement of professional growth. That raised the question as to whether growth should be measured predominantly by the practitioner targeted performance behaviors or should it equally incorporate the consultant, educator, and researcher behaviors. From the comments, it was clear that several APPs felt that their practitioner behaviors should be weighed more heavily.

BOX 2. Sample comments from PEAC Toolq evaluation surveys APP staff feedback  It was easy to match performance, projects and committee work to the categories on the tool.  Daily role (patient care) should have a larger part in the evaluation process than the grid allows.  If you are an experienced PNP, how much more growth?  [The tool] made evaluations more objective, but the majority of what we do is not measurable. Management feedback  Easy to use to evaluate staff performance.  Was an objective measure of staff performance.  Was applicable to the specific area(s) that you manage.  Made the job of evaluating staff easier.  Should undergo inter-rater reliability testing for consistency.  Good guide for evaluation, but does not capture all the behaviors.  Measures accomplishments.  As a compliment tool, helpful shows staff options to which they can aspire.

200

Volume 27  Number 3

Journal of Pediatric Health Care

There are a number of limitations to the general use and effectiveness of the PEAC Toolq, as was discovered during the evaluation phase. There was a challenge to modify an existing evaluation grid into an accurate tool for evaluating NP, CNS, and PA roles in a variety of practice settings. In order to identify practice behaviors from each setting, the tool became more generalized. The development of SMART goals facilitated the individualization of the evaluation process for each person, making it less generic. Finally, a completely objective rating scale or behavioral tool is difficult to formulate because human behavior is not objective and thus must be considered when evaluating staff performance. As APP practice continues to expand within the hospital setting, the PEAC Toolq format can be utilized by institutions as a starting place for consistent evaluation of individualÕs practice. Additionally, manager inter-rater and intra-rater reliability has not been established. In summary, managers reported the PEAC Toolq as a much more objective method of evaluating employees that have historically been inherently subjective and narrowly focused. Based on feedback received consideration will be given to higher weighting for the practitioner role than researcher, educator and consultant roles and expanding these categories to better represent practice. By the creation of this tool, through continuing revision and further enhancement, the APS managers now have a valuable resource as an objective professional development evaluation tool. There is a plan to further refine the evolving PEAC Toolq and reevaluate using the previous staff and manager surveys. Finally, a committee has been formed to create a professional practice model for advancement based on the PEAC Toolq. We would like to thank Joe Don Cavender, MSN, RN, CPNP-PC, and Judy LeFlore PhD, RN, NNP-BC, CPNP-

www.jpedhc.org

PC/AC, for their thoughtful critique and contributions in preparation of this paper.

REFERENCES Accreditation Council for Graduate Medical Education. (2011). ACGME Duty Hours: 2011 Standards. Retrieved from http:// www.acgme.org/acwebsite/dutyhours American Association of Colleges of Nursing. (2011). APRN Consensus Model. Retrieved from https://www.aacn.nche.edu/ education/apn.htm Benner, P. (2001). From Novice to Expert: Excellence and power in clinical nursing practice. Upper Saddle River, New Jersey: Prentice Hall Health. Black, A. (2001). How to Measure Staff Performance. Review of Optometry, 131(8). Capko, J. (2003). 5 Steps to a Performance Evaluation System. Family Practice Management, 10(3), 43-48. Crumbie, A., & Kyle, L. (2006). Nurse Partnership: The challenge of appraisal. Primary Health Care, 16(8), 14-16. Dreyfus S. E., & Dreyfus, H. L. (February 1980). A five-stage model of mental activities involved in direct skill acquisition. (Rep. No. ORC-80-2). University of California, Berkeley, Operations Research Center. Fletcher, J. (2004). Appraisal systems: friend or foe? Primary Health Care, 14(1), 20-21. Gentry, M. (2006). Registered nurse peer evaluation in the perioperative setting. AORN J, 84(3), 462-470. Kenny, K. J., Bake, L., Lanzon, M., Stevens, L. R., & Yancy, M. (2008). An innovative approach to peer review for the advanced practice nurse: A focus on critical incidents. Journal of American Academy of Nurse Practitioners, 20(7), 376-381. King, J. (2002). 360 Degree appraisal. British Medical Journal, 324, S195-S196. Morgan Roberts, L., Spretizer, G., Dutton, J., Quinn, R., Heaphy, E., & Barker, B. (2005). How to play to your strengths. Retrieved from http://hbr.org/product/how-to-play-to-your-strengths/ an/R0501G-PDF-ENG National League for Nursing. (1979). Position on quality review of health care and professional services review organizations (Publication No. 11-1785). New York: Author. Pearce, C. (2007). Ten steps to conducting appraisals. Nurse Management, 14(6), 21.

May/June 2013

201