Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice

G Model COLEGN-655; No. of Pages 9

ARTICLE IN PRESS Collegian xxx (2019) xxx–xxx

Contents lists available at ScienceDirect

Collegian journal homepage: www.elsevier.com/locate/coll

Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice Christine Ossenberg a,d,∗ , Marion Mitchell a,b,c , Amanda Henderson a,d a

Griffith University, School of Nursing and Midwifery, Nathan, Qld, 4111, Australia Menzies Health Institute Queensland, Griffith University, Nathan, Qld, 4111, Australia c Princess Alexandra Hospital Intensive Care Unit, Princess Alexandra Hospital, Ipswich Road, Woolloongabba, Qld, 4111, Australia d Nursing Practice Development Unit, Princess Alexandra Hospital, Ipswich Road, Woolloongabba, Qld, 4102, Australia b

a r t i c l e

i n f o

Article history: Received 8 April 2019 Received in revised form 9 October 2019 Accepted 16 October 2019 Available online xxx Keywords: Nursing Workplace-based assessment Performance Practice standards Tool Validation

a b s t r a c t Background: In Australia, performance of nursing students during clinical placements is assessed against national practice standards. Access to a tool that has a common assessment language can reduce confusion across assessors, practice areas and programmes of nursing. Aim: To revise the existing Australian Nursing Standards Assessment Tool (ANSAT) to reflect the 2016 Nursing and Midwifery Board of Australia (NMBA) registered nurse standards for practice and investigate validity of the revised tool. Design: Non-experimental study involving tool development and psychometric testing. Participants/setting: Registered nurses experienced in assessment of students from academic and clinical settings participated in the first phase of study; ANSAT scores of students across three universities were used for the second phase. Methods: A two-phased study was conducted. Phase one involved tool revision and content validation using a modified Delphi approach. Phase two involved validation of the revised tool used to assess student performance on clinical placement. Results: A 23-item tool was developed in phase one from input of eight experts, with the content validity ≥.83. The phase two field test analysed 7696 completed tools. Exploratory factor analysis extracted one factor labelled professional practice performance. The revised ANSAT demonstrated high internal consistency (Cronbach’s alpha = .985). Comparison of mean total ANSAT scores determined the tool is sensitive to differing levels of performance across year levels, placement length, and sites with statistical significance observed at p < .001. Conclusions: The revised ANSAT is a valid tool with high internal consistency that can be used to measure performance against NMBA registered nurse standards for practice. © 2019 Australian College of Nursing Ltd. Published by Elsevier Ltd.

1. Introduction Despite the changing landscape of health care resulting from digital and technological transformations (Herrman et al., 2018), nursing remains a largely practice based discipline. As such, assessment of performance in authentic practice settings is a fundamental feature in Work Integrated Learning (WIL) programmes. Workplace-based assessment (WBA) of student performance in these ‘real-world’ settings have been described as “. . . the poten-

∗ Corresponding author at: School of Nursing and Midwifery, Griffith University, Kessels Road, Nathan, 4111, Queensland, Australia. E-mail address: christine.ossenberg@griffithuni.edu.au (C. Ossenberg).

tially best way of collecting data and providing feedback on what trainees actually do in day-to-day practice” (Govaerts, van de Wiel, Schuwirth, van der Vleuten, & Muijtjens, 2013, p. 375). This is important as performance in the classroom may differ significantly from in the workplace (Jackson, 2018). Additionally, student progress in their course or entry into their chosen profession is dependent on their performance in authentic practice setting resulting in a growing emphasis on WBA (Govaerts et al., 2013). Assessment of clinical performance must demonstrate sufficient reliability and validity (Baartman, Bastiaens, Kirschner, van der Vle, & uten, 2007). The quality, utility, and acceptability of workplacebased assessment is important (Massie & Ali, 2016). The concept of validity of WBA tools is contentious with ongoing debate over reductionist approaches versus interpretive approaches (Govaerts

https://doi.org/10.1016/j.colegn.2019.10.005 1322-7696/© 2019 Australian College of Nursing Ltd. Published by Elsevier Ltd.

Please cite this article in press as: Ossenberg, C., et al. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian (2019), https://doi.org/10.1016/j.colegn.2019.10.005

G Model COLEGN-655; No. of Pages 9

ARTICLE IN PRESS C. Ossenberg et al. / Collegian xxx (2019) xxx–xxx

2

Summary of Relevance Problem In Australia, registered nurse accreditation standards require the use of validated instruments for assessment of workplace experience to evaluate student performance. What is already known The Australian Nursing Standards Assessment Tool (ANSAT) developed in 2013 responded to the need for a valid, versatile, and user-friendly tool to assess performance against the 2006 national competency standards for registered nurses. What this paper adds Evidence of both face and content validity and high internal consistency of a revised ANSAT framed by the current Australian registered nurse standards for practice. Preliminary findings indicate an ability of the ANSAT to discriminate trends in assessment of student performance.

& van der Vleuten, 2013; Govaerts et al., 2013). Irrespective of the approach, there is consensus that valid tools should indicate the evaluation of performance is credible and defensible, based on trustworthy evidence (Govaerts & van der Vleuten, 2013). Sourcing quality evidence is a vital feature of any assessment. Assessment of workplace performance relies on the evaluative judgement of an assessor against standards. Rater judgement is a known threat to the reliability of WBA and can result in large differences between performance ratings (Govaerts et al., 2013). However, test characteristics, administration conditions, and within-person differences in performance also effect the reliability of WBA (Streiner, Norman, & Cairney, 2015). Use of narrative comments to provide evidence to support the performance rating assists in transparency and reliability of assessor decision (Govaerts & van der Vleuten, 2013). Reliability can also be enhanced using standard criteria (McNamara, 2013).

competence with high internal consistency. Usability and acceptability of the tool was also demonstrated through post pilot study evaluation and think aloud interviews (Ossenberg et al., 2016). Changes to Australian legislation in July 2010 and the introduction of national registration for health professionals (which includes nurses) led to a review of the 2006 NMBA standards and subsequent development of the inaugural Australian registered nurse standards for practice (Cashin et al., 2017; Nursing & Midwifery Board of Australia, 2016). The new standards came into effect in June 2016, requiring modification of the ANSAT to reflect the changes outlined in the updated registered nurse standards for practice. 3. The study 3.1. Aim The aim of this study was to validate a revised ANSAT as a tool to assess performance in accordance with professional standards. In doing so, establish whether assessors are readily able to shift their focus in determining nurses’ performance against defined standards of practice. 3.2. Method A non-experimental, two phased approach was used to revise and validate the ANSAT. Phase one involved revision of the items and behavioural cues of the current ANSAT to meet the updated NMBA registered nurse standards for practice and review by an expert panel. Phase two involved a field test to determine psychometric properties of the revised ANSAT. Three sites in Australia that provide undergraduate nursing programmes participated in the study and were invited based on their existing use of the ANSAT. 3.3. Ethics Ethics approval was obtained (Reference number: 2016/264, approved 11 October 2016) and informed consent was collected from participants prior to study commencement. Approval was also received from participating sites. Data collected was anonymous and therefore non-identifiable to the research team.

2. Background 3.4. Phase one – expert content review Nursing programmes in Australia have diverse curricula, yet all programmes must map learning outcomes and assessments against the Nursing and Midwifery Board of Australia (NMBA) practice standards (Australian Nursing & Midwifery Accreditation Council, 2012). Nursing students are also required to demonstrate achievement of performance against these prescribed standards. Access to a tool with a common assessment language provides clarity and consistency in the attainment of standards (Ossenberg, Dalton, & Henderson, 2016) reducing confusion across assessors, practice areas and programmes of nursing. The Australian Nursing Standards Assessment Tool (ANSAT) was developed in response to the need for a versatile and user-friendly tool to assess performance against the professional standards required for entry into the nursing profession. It is used for both formative (i.e. mid-placement) assessment to assist in the identification of learning needs and summative (end-placement) assessment of student performance in the workplace environment. The ANSAT form includes space for assessors to provide evidence of how the standards were met, and behavioural cues to support assessor judgement. The original tool (based on the 2006 NMBA standards) was tested in a pilot study in 2013. Results from the pilot study demonstrated that the ANSAT was a valid tool to assess professional practice

The ANSAT 2016 was evaluated for aspects of validation including face and content validation. A modified Delphi approach was used to gain consensus from content experts to determine face validity and calculate the content validity index (CVI) of both items (I-CVI) and scale (S-CVI). Initial ‘mapping’ of the 2006 NMBA standards to the 2016 NMBA standards was undertaken by the research team. Members of the research team independently mapped existing ANSAT items to each of the 2016 standards and constructed additional items where core aspects were not represented. The majority of existing assessment items were relevant and appropriate for use with minor rewording to reflect the language of 2016 standards. Once assessment items were revised, the research team reviewed the behavioural cues to ensure alignment with proposed items and updated standards. The proposed assessment items and behavioural cues were reviewed by a local expert reference group (three nurse educators familiar with assessment of practice against nursing standards and one clinical nurse consultant specialising in evidence-based practice) to ensure all professional standard concepts were represented. This group also checked formatting, grammar, and functionality of the survey. Following this process, 22 preliminary items were included in the revised tool.

Please cite this article in press as: Ossenberg, C., et al. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian (2019), https://doi.org/10.1016/j.colegn.2019.10.005

G Model COLEGN-655; No. of Pages 9

ARTICLE IN PRESS C. Ossenberg et al. / Collegian xxx (2019) xxx–xxx

3.4.1. Data collection and analysis Due to the geographical spread of experts, an online survey was created for ease of distribution to expert panel members. Twelve experts were invited to participate in the study. Experts were asked to rate the 22 items for both relevance and clarity using a four-point Likert scale (1 = not relevant/clear; 2 = somewhat relevant/clear; 3 = quite relevant/clear; 4 = highly relevant/clear). Experts were provided an opportunity to write comments pertaining to any of the items corresponding to each of the standards. These comments were used to refine any lower rated items prior to subsequent iteration, resulting in 23 items for review for round two. As with round one, experts rated all items for relevance and clarity. Data collected from the expert panel was used to compute content validity of the items. It is widely accepted that an I-CVI > .78 and an S-CVI/Ave > .90 indicate excellent content validity (Lynn, 1986; Polit, Beck, & Owen, 2007). 3.4.2. Results Eight of the 12 experts responded to the invitation to participate in the first round. Six of the eight experts from round one participated in round two. Members of the expert panel represented five of the eight states and territories of Australia. Most participants worked in academic settings (round one, n = 7; round two, n = 5) with only one participant working in a health care setting. On average, experts in round one had 16 years’ experience coordinating, assessing, and supervising students and 17 years in round two. The average length of experience as a registered nurse was 25 years for round one and 28 years for round two. The most frequent reported age of participants in both rounds was 50–59 years, followed by 30–39 years age group. Twenty items from the preliminary 22-item pool for round one had an I-CVI ≥ .88 for relevance, while only four items had an I-CVI ≥ .88 for clarity. After refining the items based on expert comments, the online survey was redistributed for round two. Of the proposed 23 items, all items demonstrated an I-CVI ≥ .83 for both relevance and clarity. The S-CVI/Ave increased from .85 for round one to .96 for round two (refer to Supplemental material 1). The reviewed behavioural cues were distributed to members of the expert panel for consideration at the end of round two. As no comments about cues were received from any of the expert panel members, it was agreed that the cues were appropriate (see Supplemental material 2). Modification of the numerical rating scale was made before field testing in response to feedback from key stakeholders using the existing ANSAT. This changed the numerical scale from a four-point Likert scale to a five-point Likert scale to facilitate greater discrimination of performance at the lower (1 = expected behaviours and practices not performed) and higher (5 = expected behaviours and practices performed at an excellent standard) ends of the rating scale. The score of three (expected behaviours and practices performed at a satisfactory/pass standard) became the passing score for each item. The rating scale anchors/labels were also updated to reflect the change of language from competency standards to practice standards (Cashin et al., 2017). The revised ANSAT is presented in Fig. 1. No changes were made to the global rating scale (GRS). As the current ANSAT was in use by other providers, the research team agreed to name the revised tool the ‘ANSAT 2016’ for the purpose of field testing. 3.5. Phase two - field testing of the ANSAT 2016 3.5.1. Design, study site and sample A quantitative, non-experimental design was used to investigate the validity of the measurement properties of the ANSAT 2016. Convenience sampling was used for this phase of the study to source completed ANSAT 2016 scoring sheets for analysis. Three sites from across Australia participated in this phase of the study: one regional

3

site (Site A) and two metropolitan sites (Site B, Site C). These sites were early adopters of the ANSAT to measure student workplace performance. 3.5.2. Data collection Data collection occurred from July to December 2016 to correspond with clinical placements of each of the participating sites. Clinical facilitators at each participating site used the revised ANSAT 2016 to complete both a mid-placement and end-placement assessment of nursing student performance in the clinical setting. Data from completed scoring sheets were de-identified by an independent research assistant at each site and entered into an electronic spreadsheet prior to being sent to the research team and was therefore non-identifiable. 3.5.3. Data analysis Data were entered into the International Business Machines Corporation Statistical Package for Social Sciences (IBM-SPSS version 25) and screened for errors, such as anomalous and/or missing values, prior to analysis. Demographic data were analysed using descriptive statistics. Data were coded as missing when no numerical value had been assigned to the assessment item or where ‘not assessed’ was selected (as this option has no numerical value). Missing values analysis was also performed to ascertain patterns of missing data. Construct validation was assessed using Explanatory Factor Analysis (EFA) to determine if the ANSAT 2016 is a valid measure of undergraduate nursing student performance during clinical placement. EFA was performed as new items were generated and it was uncertain how the new items would be influenced by the construct of interest (Fabrigar & Wegener, 2011). Inspection of the correlation matrix and analysis of the Kaiser-Meyer-Olkin (KMO) test and Bartlett’s test of sphericity were undertaken to assess factorability of the ANSAT 2016. Known group validity was tested using a one-way analysis of variance (ANOVA) to examine if the ANSAT 2016 could differentiate performance between participating groups, indicated by comparison of the aggregate/total score. Cronbach’s alpha coefficient was used to evaluate internal consistency of the ANSAT 2016 items. 4. Results A total of 7696 ANSAT 2016 forms were completed by multiple assessors of undergraduate nursing students from three sites (Site A, n = 502, 6.5%; Site B, n = 3578, 46.5%; Site C, n = 3616, 47%) across first, second, and third year cohorts. Placement lengths varied from two to six weeks, with formative and summative assessments recorded. Placement types included acute, non-acute, mental health, and aged care and rehabilitation settings (see Table 1). 4.1. Missing values analysis Of the 7696 cases, six cases demonstrated data that were not assigned any score across the 23 items. This represents <0.1 % for all completed scoring sheets and were removed from the dataset prior to analysis. From the remaining 7690 cases, 328 cases (4.3 %) showed missing data on formative assessments and 256 cases (3.3 %) had missing data on summative assessments related to the ‘not assessed’ criteria. This represents 590 cases (7.7 %) with ‘not assessed’ recorded. Site A (n = 502) recorded 16 cases, Site B (n = 3578) recorded one case and Site C (n = 3616) recorded 573 cases with data recorded in the ‘not assessed’ criteria. A missing values analysis was undertaken using Little’s MCAR Test to ascertain patterns of missing data. Results identified that

Please cite this article in press as: Ossenberg, C., et al. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian (2019), https://doi.org/10.1016/j.colegn.2019.10.005

G Model COLEGN-655; No. of Pages 9 4

ARTICLE IN PRESS C. Ossenberg et al. / Collegian xxx (2019) xxx–xxx

Fig. 1. ANSAT front page.

Please cite this article in press as: Ossenberg, C., et al. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian (2019), https://doi.org/10.1016/j.colegn.2019.10.005

G Model COLEGN-655; No. of Pages 9

ARTICLE IN PRESS C. Ossenberg et al. / Collegian xxx (2019) xxx–xxx

5

Table 1 Demographics characteristics of field test data. Site A n (%) 502 (6.5)

Site B n (%) 3578 (46.5)

Site C n (%) 3616 (47.0)

Combined n (%) 7696 (100.0)

210 (41.8) 234 (46.6) 58 (11.6)

1232 (34.4) 2333 (65.2) 13 (0.4)

30 (0.8) 1165 (32.2) 2395 (66.2) 26 (0.7)

1472 (19.1) 3732 (48.5) 2466 (32.0) 26 (0.3)

Placement length 2 weeks 3 weeks 4 weeks 6 weeks

32 (6.4) 4 (0.8) 434 (92.4) 2 (0.4)

3536 (98.8)

1832 (50.7) 1165 (32.2) 619 (17.1)

5400 (70.2) 1169 (15.2) 1125 (14.6) 2 (0.0)

Placement type Acute Non-acute Mental health Aged care and rehabilitation Missing

4 (0.8) 2 (0.4) 12 (2.4) 460 (91.6) 24 (4.8)

1493 (41.7) 555 (15.5) 588 (16.4) 872 (24.4) 70 (2.0)

2478 (68.5) 716 (19.8) 63 (1.7) 359 (10.0)

3975 (51.7) 1273 (16.5) 663 (8.6) 1691 (22.0) 94 (1.2)

Assessment type Formative (mid-placement) Summative(end-placement) Missing

247 (49.2) 249 (49.6) 6 (1.2)

1795 (50.2) 1783 (49.8)

1922 (53.2) 1694 (46.8)

3964 (51.4) 3726 (48.4) 6 (0.1)

Year level 1st year 2nd year 3rd year Missing

42 (1.2)

data were not missing at random (␹2 = 4083.16, p < .001). Analysis of the 590 cases demonstrated patterns of missing values for four items: 3.3 (n = 84, 1.1 %); 3.4 (n = 129, 1.7 %); 6.2 (n = 350, 4.5 %); and 6.3 (n = 154, 2.0 %) (refer to Fig. 1 for item description). Use of estimated maximisation is not recommended with data that is not missing at random (Tabachnick & Fidell, 2014). While ‘not assessed’ is a designated option for the assessors, it has no numerical value in relation to rating student performance. Therefore, use of a ‘best guess’ approach such as mean substitution was considered unsuitable and the decision was made to remove these cases from the data set.

explained 77.07 % (Site A), 73.26 % (Site B) and 74.24 % (Site C) of the variance respectively. Individual scree plots also supported a one factor solution as seen in the analysis of the combined data set. Table 2 outlines factor loadings and communalities for each analysis. The internal consistency was calculated with a Cronbach’s alpha coefficient (␣) for the combined data set (n = 7106) reported at ␣ = .985 (23 items). When calculated for each site, the ANSAT 2016 correspondingly demonstrated a Cronbach’s alpha of .986 (Site A), .983 (Site B) and .984 (Site C). Comparative values of alpha for each standard are presented in Table 3.

4.2. Construct validation

4.3. Known-group validation

Examination of the correlation matrix demonstrated all coefficients to be .6 and above. The Kaiser-Myer-Olkin measure of sample adequacy was .985, well above the recommended value of .6 (Pallant, 2016). Bartlett’s test of sphericity was significant at <.001. The diagonals of the anti-image correlation matrix were all higher than .9 and communalities also >.6, therefore indicating that the items of the ANSAT were suitable for factor analysis (Pett, Lackey, & Sullivan, 2003). Exploratory factor analysis using maximum likelihood extraction was conducted on the combined data set to determine the factor structure of the ANSAT 2016 (Tabachnick & Fidell, 2014). Initial analysis showed that one factor had an eigenvalue over Kaiser’s criterion of 1 and in combination explained 75.26 % of the variance; indicating that items in the ANSAT 2016 strongly measure one concept labelled professional practice performance. Examination of the scree plot (Fig. 2) demonstrated a ‘levelling-off’ after one factor. Extraction of one factor is also supported by the results of Monte Carlo simulation (parallel analysis), which demonstrated only one value exceeding corresponding criterion values for a randomly generated data set (Supplemental material 3). Both orthogonal (varimax) and oblique (oblimin) solutions were examined with little difference between either solution evident. The decision to use a varimax rotation for the final solution was made to minimise the number of vairables that have high loadings on each other (Pallant, 2016). Exploratory factor analysis was repeated for each site to determine potential differences between groups. Initial eigenvalues

Total ANSAT 2016 scores were subjected to a series of factorial analyses of variance (ANOVA) to compare mean total ANSAT 2016 scores for the variables of site, year level and placement length. Statistical significance was determined at the .05 level. The first three by three factorial ANOVA examined influence of the site (Site A, Site B, and Site C) and student year level (first year, second year, and third year) on the mean total ANSAT 2016 score. There was a significant main effect of site, F(2, 7082) = 20.87, p < .001 ␩2 = .006. Post hoc analysis using Scheffe indicated that the mean total ANSAT score was significantly higher for Site A (M = 89.92, SD = 16.19) and Site C (M = 89.27, SD = 14.96) than Site B (M = 81.63, SD = 13.45). There was also a significant main effect of year level, F(2, 7082) = 19.40, p < .001, ␩2 = .005, indicating that the mean total ANSAT 2016 score for third year nursing students (M = 90.19, SD = 14.82) was higher than second year nursing students (M = 84.91, SD = 14.69) and first year nursing students (M = 80.13, SD = 12.88). The interaction effect between site and year level was also significant, F(4, 7082) = 8.45, p < .001, ␩2 = .005 indicating that the effect of year level on total ANSAT 2016 scores was observed across each site. The second three by three factorial ANOVA examined the relationship of student year level and placement length (two weeks, three weeks, and four or more weeks) on mean total ANSAT 2016 scores. The main effect for placement length was statistically significant, F(2, 7083) = 93.02, p < .001, ␩2 = .026. Similarly, the interaction of year level and placement length on mean total ANSAT 2016 scores was also significant, F(3, 7083) = 6.92, p < .001, ␩2 = .003, indi-

Please cite this article in press as: Ossenberg, C., et al. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian (2019), https://doi.org/10.1016/j.colegn.2019.10.005

1.1

Site B (n = 3577)

Site C (n = 3043)

Combined (n = 7106)

Communalities

Factor loading

Communalities

Factor loading

Communalities

Factor loading

Communalities

.860

.740

.806

.650

.831

.690

.832

.692

.875

.766

.823

.678

.856

.732

.850

.722

.838

.702

.818

.669

.792

.627

.816

.666

.872

.760

.834

.695

.845

.714

.853

.728

.871 .850

.759 .723

.856 .852

.732 .725

.863 .825

.745 .681

.868 .846

.754 .715

.877

.769

.855

.731

.859

.738

.865

.749

.869

.755

.841

.707

.869

.756

.865

.748

.872

.760

.845

.714

.838

.703

.850

.723

.825

.680

.837

.701

.845

.713

.851

.723

.835

.697

.846

.715

.853

.727

.856

.733

.887

.786

.841

.708

.859

.738

.862

.742

.887

.787

.849

.721

.862

.743

.864

.747

.887

.786

.871

.759

.894

.799

.890

.791

.858

.736

.852

.726

.856

.733

.862

.743

.874

.763

.853

.727

.862

.744

.866

.750

.885

.783

.875

.766

.868

.753

.881

.776

.883

.779

.862

.743

.876

.767

.879

.772

.879

.773

.873

.762

.875

.766

.881

.776

.885

.783

.834

.695

.814

.663

.830

.689

.872

.761

.873

.762

.859

.738

.872

.760

.907

.822

.873

.763

.881

.777

.888

.788

.902

.813

.847

.718

.872

.760

.872

.760

17.73 77.07 .979 < .001

16.85 73.26 .983 < .001

17.08 74.24 .984 < .001

17.31 75.26 .985 < .001

ARTICLE IN PRESS

Factor loading

C. Ossenberg et al. / Collegian xxx (2019) xxx–xxx

Complies and practices according to relevant legislation and local policy Uses an ethical framework to guide decision making and 1.2 practice 1.3 Demonstrates respect for individual and cultural (including Aboriginal and Torres Strait Islander) preference and differences 1.4 Sources and critically evaluates relevant literature and research evidence to deliver quality practice Maintains the use of clear and accurate documentation 1.5 Communicates effectively to maintain personal and 2.1 professional boundaries Collaborates with the health care team and others to share 2.2 knowledge that promotes person-centred care Participates as an active member of the healthcare team to 2.3 achieve optimum health outcomes Demonstrates respect for a person’s rights and wishes and 2.4 advocates on their behalf 3.1 Demonstrates commitment to life-long learning of self and others Reflects on practice and responds to feedback for 3.2 continuing professional development Demonstrates skills in health education to enable people 3.3 to make decisions and take action about their health Recognises and responds appropriately when own or 3.4 other’s capability for practice is impaired 3.5 Demonstrates accountability for decisions and actions appropriate to their role Completes comprehensive and systematic assessments 4.1 using appropriate and available sources Accurately analyses and interprets assessment data to 4.2 inform practices Collaboratively constructs a plan informed by the 5.1 patient/client assessment 5.2 Plans care in partnership with individuals/significant others/health care team to achieve expected outcomes Delivers safe and effective care within their scope of 6.1 practice to meet outcomes Provides effective supervision and delegates care safely 6.2 within their role and scope of practice 6.3 Recognise and responds to practice that may be below expected organisational, legal or regulatory standards Monitors progress toward expected goals and health 7.1 outcomes Modifies plan according to evaluation of goals and 7.2 outcomes in consultation with the health care team and others Initial eigenvalues Total variance explained (%) Kaiser-Meyer-Olkin (KMO) measure of sample adequacy Bartlett’s test of sphericity

Site A (n = 486)

G Model

ITEM

COLEGN-655; No. of Pages 9

6

Please cite this article in press as: Ossenberg, C., et al. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian (2019), https://doi.org/10.1016/j.colegn.2019.10.005

Table 2 Factor loadings and communalities for exploratory factor analysis by site.

G Model COLEGN-655; No. of Pages 9

ARTICLE IN PRESS C. Ossenberg et al. / Collegian xxx (2019) xxx–xxx

7

Fig. 2. Scree plot–exploratory factor analysis.

Table 3 Cronbach’s alpha for revised ANSAT.

Total Scale – Professional Practice Performance Subscale Standard 1 – Thinks critically and analyses nursing practice Standard 2 – Engages in therapeutic and professional relationships Standard 3 – Maintains capability for practice Standard 4 - Comprehensively conducts assessments Standard 5 – Develops a plan for nursing practice Standard 6 – Provides safe, appropriate and responsive quality nursing practice Standard 7 – Evaluates outcomes to inform nursing practice

cating that the mean total ANSAT 2016 scores were greater for placements of four or more weeks (M = 92.45, SD = 14.82) than three week placements (M = 87.98, SD = 15.06) and 2 week placements (M = 83.48, SD = 14.24) for each year level.

5. Discussion The purpose of this study was to assess the validity of a revised ANSAT and explore how assessors adopted the form. This revalidation responds to limitations identified in the pilot study of using the ANSAT 2016 to assess performance of first year nursing students and a larger field test addressed in the original pilot study (Ossenberg et al., 2016). Sample adequacy increased through increased sample size (from n = 220 to n = 7696) and inclusion of data from three sites across Australia in both regional and metropolitan settings. This provides broader representation of the undergraduate nursing student population increasing generalisability and transferability of application of the ANSAT 2016. An exploratory factor analysis determined one latent factor that underlies the 23 items of the ANSAT 2016. This is reflective of the intent of the tool and the nature of the national standards for practice. The extracted factor was relabelled professional practice performance to reflect the change of language in the standards. To determine that this factor structure is supported and demonstrates construct validation, it is strongly recommended that further analysis using confirmatory factor analysis be undertaken (Cramer, 2003; Tabachnick and Fidell, 2014). Analysis has confirmed that the ANSAT 2016 consistently reflects the construct it was designed to measure (i.e. a measure

Items

Site A n = 486

Site B n = 3577

Site C n = 3043

Combined n = 7106

1.1–7.2

.986

.983

.984

.985

1.1–1.5 2.1–2.4 3.1–3.5 4.1, 4.2 5.1, 5.2 6.1–6.3 7.1, 7.2

.944 .945 .942 .940 .934 .928 .951

.927 .933 .936 .911 .921 .906 .918

.930 .933 .941 .915 .925 .894 .924

.935 .938 .942 .920 .929 .906 .929

of performance against national standards for practice in authentic practice settings). As identified, the Cronbach’s alpha for the ANSAT is ≥.95 (Table 3). The issue of what value of alpha is acceptable is still debated (Streiner et al., 2015). While it is acknowledged that an alpha that is very high (i.e. ≥ .90) may suggest item redundancy (Boyle, 1991; Taber, 2018), items in the scale underwent a rigorous content validation process by an expert panel. In accordance with expert opinion, the resulting 23 items reflect the detail required for student assessment against all aspects of the requisite standards for practice. As it is a requirement by the NMBA that all aspects of the standards must be met, removal of an item from the scale may be imprudent. Inter-rater reliability of the ANSAT 2016 has not been examined to date and is a concept that warrants further investigation. Emerging literature challenges the value of evaluating inter-rater reliability as rating outcomes are “. . .determined by a complex and interrelated set of factors in the social setting of the assessment process, such as local norms and values, time pressure, assessment goals and affective factors” (Govaerts et al., 2013, p. 376). While provision of assessor training is vital to ensure assessment processes are consistent, inter-rater consistency and learner performance cannot be guaranteed in the dynamic and diverse settings in which work integrated learning experiences occur. Exploring this view of rater reliability is beyond the scope of this study and needs further discussion and scrutiny. The ANSAT 2016 has a minimum total score of 23 and a maximum of 115. The ANSAT 2016 has capacity to derive a total score and set a ‘pass’ value. However, it is not commonly used in this format as nursing students must demonstrate ‘expected behaviours

Please cite this article in press as: Ossenberg, C., et al. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian (2019), https://doi.org/10.1016/j.colegn.2019.10.005

G Model COLEGN-655; No. of Pages 9 8

ARTICLE IN PRESS C. Ossenberg et al. / Collegian xxx (2019) xxx–xxx

and practices are performed at a satisfactory/pass standard’ for each item within the corresponding standard for performance to be recommended as meeting that specific practice standard. Total scores were calculated for the purpose of analysis. The ANOVA elicited several key findings. Firstly, nursing students at Site A and Site C have a statistically higher (p < .001) mean total score than those at Site B. Data used for analysis was provided by each site; no demographic data were provided about the assessors (e.g. highest qualification; level of assessor experience; assessor training provided) and therefore it is difficult to attribute full meaning to this finding. Access to data for possible benchmarking of student performance on clinical placement between programmes using a standardised tool such as the ANSAT would be of interest to track and compare student cohorts as well as longitudinal trends of performance (Takashima, Burmeister, Ossenberg, & Henderson, 2019). Secondly, mean total scores were higher as placement length increased for each site. It is posited that as the length of placement increases, there is more opportunity for students to consolidate theory and opportunities to practice within the clinical learning environment. Furthermore, longer placements provide assessors with additional time to collect evidence of student performance to make a professional judgement and rate student’s performance according to standards for practice. A notable observation was the number of items with ‘not assessed’ recorded on completed ANSAT 2016 scoring sheets. Little’s MCAR identified that this data was not missing at random but possibly due to system bias (Tabachnick & Fidell, 2014). This is supported by the 573 cases from Site C with ‘not assessed’ recorded. On review, the four items that had ‘not assessed’ attributed to them (items 3.3, 3.4, 6.2, 6.3) were items that highlighted the updated focus of the 2016 NMBA standards relating to capability for practice and health education. It is postulated that at the time of field testing, assessors may not have had an opportunity to develop a detailed understanding of the newer language and area of focus of the standards for practice and consequently uncertain as to how students could best demonstrate performance in these areas. While these areas were implicit in the 2006 NMBA competency standards, the updated standards for practice articulate these areas more clearly. It is recognised that ‘not assessed’ may be seen in formative assessments. In this situation, it serves as a ‘flag’ to both the student and assessor to identify opportunities for more practice or observation by the assessor. However, the presence of ‘not assessed’ in summative assessments is a concern. Recording ‘not assessed’ indicates the assessor has no evidence on how the student performed that specific item. The effects of this are twofold. Firstly, individual student progress may be impacted as programme providers must decide if the student continues or if the placement must be repeated. Secondly, at a broader programme level, higher rates of ‘not assessed’ potentially reflect such factors as assessor capability (including familiarity with assessment processes, NMBA standards) or suitability of placement for students to meet practice requirements and course goals. The updated standards for practice have been in use since June 2016; further research should be considered to determine the merit of the ‘not assessed’ criteria to the assessment process. Reliable interpretation of the ANSAT is important to ensure consistent assessment practices. While the importance of assessor training is necessary to aid understanding of workplace-based assessment has been identified, it can be difficult to offer face-toface training due to geographical spread of nursing programmes using the tool. In response, a detailed resource manual has been developed to provide students, assessors, and higher education providers with information on use of the ANSAT for assessment in authentic practice settings, explanation of rating criteria, and fre-

quently asked questions (www.ansat.com.au/home/assessmentmanual). 6. Limitations There were several limitations with this study. The ANSAT 2016 is specifically designed for the Australian context and validity has only been established for assessing performance of undergraduate nursing students. Inter-rater reliability is a consideration of the agreement or degree of correspondence between assessors’ ‘score’ when rating the same student’s performance. This has not been investigated to date and is an issue for subsequent research. Another limitation identified was the absence of information about clinical assessors as a result of using second hand data. Inclusion of this data may have helped to interpret the observed findings. 7. Conclusions The revised tool replaces the initial ANSAT. To reduce confusion amongst end-users, the ANSAT 2016 will revert to being referred to as the ANSAT. Overall, the ANSAT continues to be a robust and valid tool to measure performance against the NMBA registered nurse standards for practice with high internal consistency. Updated behavioural cues reflect current standards and serve as indicators of practice in everyday language. Statistical analysis demonstrated that the revised ANSAT reflected one factor—professional practice competence—across all data and at an individual site level. ANSAT scores indicate that student performance improves with longer workplacebased placements. Training and support for assessors on use of workplace-based assessment tools and site assessment processes are key to ensure assessors have clear expectations of assessment requirements and awareness of their possible biases. Inclusion of a feedback section also encourages assessors to provide written examples as evidence to support rating selection. The ANSAT is primarily used in pre-licensure programmes. As it is based on the registered nurse standards for practice, use of the ANSAT in the assessment of performance for registered nurses at all stages of practice is fitting. Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Ethical approval Ethics approval was obtained from Griffith University Human Research Ethics Committee (Reference number: 2016/264, approved 11 October 2016) and informed consent was collected from participants prior to study commencement. All authors of this manuscript declare: 1 this material has not been published in whole or in part elsewhere; 2 the manuscript is not currently being considered for publication in another journal; 3 all authors have been personally and actively involved in substantive work leading to the manuscript and will hold themselves jointly and individually responsible for its content. Declaration of Competing Interest None of the authors have any conflict of interest to declare for this article.

Please cite this article in press as: Ossenberg, C., et al. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian (2019), https://doi.org/10.1016/j.colegn.2019.10.005

G Model COLEGN-655; No. of Pages 9

ARTICLE IN PRESS C. Ossenberg et al. / Collegian xxx (2019) xxx–xxx

Acknowledgements This study was undertaken as part of doctoral studies supported by Metro South Health Study, Education and Research Trust Account post graduate scholarship and a Research Training Program Domestic Fee Offset scholarship provided by the Australian Government Department of Education and Training (administered by Griffith University). The authors would also like to acknowledge the generous assistance of the clinical placement offices at each site for assistance with the provision of data for analysis. Appendix A. Supplementary data Supplementary material related to this article can be found, in the online version, at doi:https://doi.org/10.1016/j.colegn.2019.10. 005. References Australian Nursing and Midwifery Accreditation Council. (2012). Registered nurse accreditation standards. Baartman, L. K. J., Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2007). Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks. Educational Research Review, 2(2), 114–129. http://dx.doi.org/10.1016/j.edurev.2007.06.001 Boyle, G. J. (1991). Does item homogeneity indicate internal consistency or item redundancy in psychometric scales? Personality and Individual Differences, 12(3), 291–294. Cashin, A., Heartfield, M., Bryce, J., Devey, L., Buckley, T., Cox, D., . . . & Fisher, M. (2017). Standards for practice for registered nurses in Australia. Collegian, 24(3), 255–266. Cramer, D. (2003). Advanced quantitative data analysis. UK): McGraw-Hill Education. Fabrigar, L. R., & Wegener, D. T. (2011). Exploratory factor analysis. Oxford University Press. Govaerts, M. J. B., & van der Vleuten, C. P. M. (2013). Validity in work-based assessment: Expanding our horizons. Medical Education, 47(12), 1164–1174. http://dx.doi.org/10.1111/medu.12289 Govaerts, M. J. B., van de Wiel, M. W. J., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education, 18(3), 375–396. http://dx.doi.org/10.1007/s10459-012-9376-x

9

Herrman, M., Boehme, P., Mondritzki, T., Ehlers, J., Kavadias, S., & Truebel, H. (2018). Digital transformation and distruption of the health care sector: Internet-based observational study. Journal of Medical Internet Research, 20(3), e104. http://dx.doi.org/10.2196/jmir.9498 Jackson, D. (2018). Challenges and strategies for assessing student workplace performance during work-integrated learning. Assessment & Evaluation in Higher Education, 43(4), 555–570. Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. Massie, J., & Ali, J. M. (2016). Workplace-based assessment: A review of user perceptions and strategies to address the identified shortcomings. Advances in Health Sciences Education, 21(2), 455–473. http://dx.doi.org/10.1007/s10459015-9614-0 McNamara, J. (2013). The challenge of assessing professional competence in work integrated learning. Assessment & Evaluation in Higher Education, 38(2), 183–197. Nursing and Midwifery Board of Australia. (2016). Registered nurse standards for practice Retrieved from. http://www.nursingmidwiferyboard.gov.au/News/ 2016-02-01-revised-standards.aspx Ossenberg, C., Dalton, M., & Henderson, A. (2016). Validation of the Australian Nursing Standards Assessment Tool (ANSAT): A pilot study. Nurse Education Today, 36(23-30), 28. http://dx.doi.org/10.1016/j.nedt.2015.07.012 Pallant, J. (2016). SPSS survival manual: A step by step guide to data analysis using IBM SPSS (6th ed.). Sydney: Allen & Unwin. Pett, M., Lackey, N., & Sullivan, J. (2003). Assessing characteristics of matrices. In Making sense of factor analysis. Thousand Oaks, California: SAGE Publications, Inc. http://dx.doi.org/10.4135/9781412984898. Retrieved from. http:// methods.sagepub.com/book/making-sense-of-factor-analysis Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. Streiner, D. L., Norman, G. R., & Cairney, J. (2015). Health measurement scales: A practical guide to their development and use (5th ed.). USA: Oxford University Press. Tabachnick, B. G., & Fidell, L. S. (2014). Using multivariate statistics (6th ed.). Harlow; United Kingdom: Pearson Education Limited. Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48(6), 1273–1296. Takashima, M., Burmeister, E., Ossenberg, C., & Henderson, A. (2019). Assessment of the clinical performance of Nursing students in the workplace: Exploring the role of benchmarking using the Australian Nursing Standards Assessment Tool (ANSAT). Collegian, 26(4), 502–506. http://dx.doi.org/10.1016/j.colegn. 2019.01.005. Advanced online publication

Please cite this article in press as: Ossenberg, C., et al. Adoption of new practice standards in nursing: Revalidation of a tool to measure performance using the Australian registered nurse standards for practice. Collegian (2019), https://doi.org/10.1016/j.colegn.2019.10.005