- Email: [email protected]
Development of a simulation-based assessment to evaluate the clinical competencies of Korean nursing students
Nurse Education Today 36 (2016) 337–341
Contents lists available at ScienceDirect
Nurse Education Today journal homepage: www.elsevier.com/nedt
Development of a simulation-based assessment to evaluate the clinical competencies of Korean nursing students☆ Kyongok Park a, Youngmee Ahn b, Narae Kang b, Min Sohn b,⁎ a b
Department of Nursing, Far East University, Eumseong, Republic of Korea Department of Nursing, Inha University, Incheon, Republic of Korea
a r t i c l e
i n f o
Article history: Accepted 21 August 2015 Keywords: Asthma Diabetes Global rating scale Nursing students Simulation
s u m m a r y Objectives: To describe a simulation-based assessment (SBA) to evaluate the clinical competencies of nursing students in children's health and to compare its results with grade point average (GPA), self-efficacy, topic-specific knowledge, and self-reported clinical competency using the Six-D Scale. Methods: This cross-sectional, descriptive study recruited nursing students from a children's health clinical practicum. Students were assigned to either an asthma (n = 55) or a type 1 diabetes (n = 48) care scenario conducted on a high-fidelity simulator. Clinical competencies were assessed using the global rating scale (GRS) and a checklist. Results: Data on 103 students were analyzed. The SBA-GRS indicated that 64.6%–87.3% of students passed. The SBA-GRS showed a statistically significant positive association with the SBA checklist in both the asthma (rho = .763, p b .001) and the type 1 diabetes (rho = .475, p = .001) group. In the asthma group, the SBA-GRS and checklist showed statistically significant associations with GPA (rho = .413, p = .002 vs. r = .508, p b .001) and the Six-D Scale (rho = .266, p = .049 vs. r = .352, p = .008); in the diabetes group, only the SBA checklist showed a statistically significant association with self-efficacy (r = .339, p = .018) and the Six-D Scale (r = .373, p = .009). Four groups by SBA-GRS had statistically significant differences in scores on the SBA checklist in both groups (F = 25.757, p b .001 in the asthma group; F = 4.790, p = .006 in the diabetes group) and GPA only in the asthma groups (F = 6.095, p b .001). Conclusion: SBA was found to be feasible for nursing students. The GRS and checklist were reasonably correlated with other evaluation methods of student competency, but correlations were better with easier scenarios. © 2015 Elsevier Ltd. All rights reserved.
Introduction The assessment of clinical competency has always been a key component of clinical education for health professionals. In addition to traditional methods such as written tests and clinical observations, direct evaluation methods conducted in standard environments have attracted the attention of nursing educators. For example, objective structured clinical examination (OSCE) has been widely used in many health professions programs, including nursing. OSCE has been shown to be reliable and valid in assessing medical students (Dong et al., 2014), nursing students (Mitchell et al., 2009), and postgraduate professionals (Schoenmakers & Wens, 2014). However, adapting OSCE can be
☆ This study was supported by the Basic Science Research Program of the National Research Foundation of Korea and funded by the Ministry of Education, Science, and Technology (2011-0009627). ⁎ Corresponding author at: Department of Nursing, Inha University, 100 Inharo, NamGu, Incheon, South Korea. Tel.: +82 32 860 8212; fax: +82 32 874 5880. E-mail address: [email protected] (M. Sohn).
http://dx.doi.org/10.1016/j.nedt.2015.08.020 0260-6917/© 2015 Elsevier Ltd. All rights reserved.
problematic, particularly when resources are limited. OSCE requires multiple clinical scenarios and space, equipment, supplies, standardized patients, and trained examiners to evaluate multiple observations of individual students (McIlroy et al., 2002). Because these requirements are so expensive (Palese et al., 2012), OSCE is not available in many Korean nursing programs. Nonetheless, standardized objective assessment of clinical competency is still an important part of nursing education, and simulationbased assessment (SBA) could be a useful alternative. Not only has simulation been widely used for the clinical education of nursing students and nurses but it has also been used to assess the clinical competencies of surgeons (Jaffer et al., 2015), medical residents (Burns et al., 2013; Fehr et al., 2011), and nurse anesthetists (Henrichs et al., 2009). Compared with OSCE, SBA also has unique benefits for student evaluation: its operating costs are smaller, it is less demanding of examiners, and it provides students with almost unlimited opportunities to practice. The most commonly used evaluation tool in OSCE or SBA of students is the checklist. A checklist consists of a series of items that are usually rated dichotomously: 1 (“performed”) or 0 (“not performed”). It
338
K. Park et al. / Nurse Education Today 36 (2016) 337–341
facilitates a thorough assessment of inexperienced examinees at every stage of performance. In addition, educators often use checklist results to advise students how they performed on individual items. However, checklists are not always the best option. They are case-specific evaluation tools, and their items are scenario-dependent, which is highly demanding for both developers and examiners. Furthermore, previous studies have concluded that checklists make it difficult to evaluate higher levels of performance such as integrity, prioritization, and efficiency (Hodges & McIlroy, 2003). The global rating scale (GRS) can also be used to assess clinical competency. The GRS typically requires examiners to rate the overall performance of examinees based on expert judgment and often uses only a few items, on occasions even a single item (Hodges & McIlroy, 2003). Global rating scales have several benefits over checklists: They are easily scored, allow greater flexibility in judgment, take less time to administer, and provide a better means of assessing improvements, from the novice to the expert level (Kim et al., 2009). In addition, the GRS seems to capture aspects (e.g., empathy, communication, and short and effective skills) that are not well-addressed in textbooks but are usually found in clinical experts. Moreover, a well-developed GRS affords examiners the opportunity to test examinees in real clinical settings. In SBA, either the GRS or checklist can be used to assess student competencies. Ilgen et al. (2015) recently reported that both methods had good reliability and validity when used with SBA (Ilgen et al., 2015). However, the evidence is still limited, particularly in nursing education. The association between these two methods and other types of assessment of student competencies during nursing programs is lacking. For example, grade point average (GPA), self-efficacy, knowledge of the target topic, and self-reported competency, as determined by the Six-Dimension Scale of Nursing Performance (Six-D Scale), have been used in previous studies of OSCE of students (Eftekhar et al., 2012; Park et al., 2013), but these have rarely been investigated in conjunction with SBA. This study, therefore, was undertaken to develop an SBA for nursing students and to compare the efficacy of a GRS and a checklist to measure student competency with other methods of evaluation. We were particularly interested in children's health. The clinical practicum of nursing education for children has been made difficult by a decreasing number of clients and clinical placements. This means that students have fewer opportunities to practice nursing care, and educators are more pressured to ensure that students can perform well in real care situations. For these reasons, children's health is one area where simulation is widely used for the clinical education of nursing students (Hayden, 2010). For this study, we selected asthma and type 1 diabetes, two chronic health conditions that are common in children. Our specific aims were (1) to describe the development of an SBA using a GRS and checklist to assess the clinical competencies of nursing students for children with asthma or diabetes and (2) to compare SBA results with different types of competency tests such as GPA, self-efficacy, knowledge of the topics, and the Six-D Scale. Methods Study Design and Participants This was a descriptive, cross-sectional study. Data were collected from March 2012 to June 2013. The study participants were conveniently recruited from a group of fourth year nursing students who were participating in a clinical practicum on children's health in a university located in Incheon, South Korea. Development of SBA Modules for Asthma and Type 1 Diabetes Two SBA modules, one for asthma and the other for type 1 diabetes, were developed by a focus group, which included two nursing faculty
and two nursing researchers in children's health and nursing simulation education. After the modules had been drafted, they were reviewed by clinicians (i.e., a pediatrician, a clinical nurse specialist, and a nurse manager in a pediatric department) and subsequently revised based on their feedback. Scenarios on asthma and diabetes were developed with two levels of difficulty to evaluate their effect on study outcomes. The principals of both scenarios were young teenagers. The asthma scenario involved a teenager who visited an emergency department with shortness of breath; the diabetes scenario involved an inpatient with a cold who was admitted for high blood sugar. The diabetes scenario was the more difficult scenario because the module presented at least two nursing problems (i.e., high blood sugar and dehydration), while the asthma scenario involved only an acute exacerbation of asthma. Both modules comprised a planning phase (15 min), a simulation performance phase (15 min), and a documentation/survey completion phase (15 min). Each student performed one of two scenarios on the first day of the 5-day practicum. The scenario was given alternatively among the two scenarios per week. Students were provided with written information that included a summary of each scenario, the physician's order, medication administration records, nursing records, drug information, and a list of actions that the students were required to perform. These actions included assessment, interventions, health promotion counseling, and presentation of an appropriate professional attitude. After students had finished planning, they proceeded to the simulation phase. While an operator controlled the simulator, an examiner observed the students through a one-way mirror and recorded their performances.
Measurements We collected data on basic characteristics and six measures of competencies. The basic characteristics consisted of age, gender, experience of simulation, and amount of previous simulation use. The six measures of competencies were SBA-GRS and checklist scores, GPA, knowledge about asthma or diabetes, self-efficacy, and Six-D Scale scores. GPA, the average grade of six semesters, was obtained from the department of nursing. The SBA-GRS and checklist, questionnaires on asthma and diabetes knowledge, and self-efficacy were developed by the focus group and reviewed by clinicians, as described above.
SBA-GRS Researchers may use 3- to 5-point Likert scales in a GRS (Liddle, 2014). We selected a 4-point Likert scale to avoid examiners selecting, perhaps unconsciously, a middle grade. Based on their performance, students received one of four grades: “very good pass,” “clear pass,” “borderline pass,” and “fail.”
SBA Checklist The asthma scenario checklist comprised five domains (22 items): assessment by interview (7 items), physical examination (4 items), intervention (4 items), health education (2 items), and professional attitude (5 items). Each item was coded either 1 (“correctly performed”) or 0 (“not correctly performed”). Domain scores were expressed as percentages of total possible scores. Total checklist scores were defined as the average scores of the five domains, which meant that domains were equally weighted. The diabetes scenario checklist also consisted of five domains, but it had 23 items: assessment by interview (9 items), physical examination (3 items), intervention (3 items), health education (3 items), and professional attitude (5 items). The same scoring system was applied to this scenario. Inter-rater reliability of the checklists was 0.75 for the asthma scenario and 0.87 for the diabetes scenario.
K. Park et al. / Nurse Education Today 36 (2016) 337–341 Table 1 Characteristics of nursing students by scenario (N = 103). Student groups Asthma (n = 55)
t-test or χ2 (p)
Diabetes (n = 48)
Frequency (%) or mean (SD) Age (year) 22.0 (1.1) 21.5 (0.7) 2.74 (.007) Experience with simulation 29 (52.7) 26 (54.2) 0.021 (N.999) Length of simulation experience 1.5 (0.9) 1.8 (1.2) −1.278 (.208) among experienced students (hours) SBA-GRS, Pass 48 (87.3) 31 (64.6) 7.383 (.007) Very good pass 4 (7.3) 1 (2.1) Clear pass 18 (32.7) 6 (12.5) Borderline pass 26 (47.3) 24 (50.0) Clear fail 7 (12.7) 17 (35.4) SBA Checklist 49.0 (15.3) 39.5 (13.0) 3.370 (.001) GPA 3.5 (0.4) 3.6 (0.4) −1.227 (.223) Knowledge 8.0 (1.4) 7.9 (1.4) .530 (.597) Self-efficacy 6.8 (1.3) 6.8 (1.2) −.305 (.761) Six-D Scale 2.2 (0.4) 2.1 (0.5) 1.987 (.050) SD = standard deviation; SBA = simulation-based assessment; GRS = global rating scale; GPA = grade point average; Six-D Scale = Six-Dimension Scale of Nursing Performance.
Knowledge Knowledge about asthma and diabetes was assessed using a 10-item quiz, which included questions on assessment, intervention, medication, and health promotion for children with asthma or type 1 diabetes. Scores for total knowledge were based on the number of items students answered correctly. Self-efficacy Based on Bandura’s (1997) social cognitive theory, we developed the instrument to examine how confident students were in providing nursing care to children with asthma or diabetes. The scope of that care involved assessment, prioritizing nursing problems, intervention, and health promotion education. Self-efficacy in nursing care for children with asthma or diabetes included 30 and 32 items, respectively. Students graded their confidence levels from 0 (“not at all confident”) to 10 (“absolutely confident”). Total scores were defined as the mean of all items and ranged from 0 to 10. Higher scores indicated higher levels of confidence. Cronbach's alpha was 0.945 for asthma and 0.959 for diabetes. The Six-D Scale Originally developed to evaluate the competency of nurses by assessing their level of confidence for selected nursing actions (Schwirian, 1978), the Six-D Scale has also been used for nursing students (Klein & Fowles, 2009). The scale comprises six subscales
339
(52 items): leadership (5 items), critical care (7 items), teaching and collaboration (11 items), planning and evaluation (7 items), and interpersonal relations and communication (12 items). Students were asked to rate their competencies from 1 (“not very well”) to 4 (“very well”). Total scores were defined as the mean item score; higher scores indicated greater nursing competency. We used a Korean version of the Six-D Scale, which was developed and tested on Korean nursing students (Park et al., 2013). Cronbach's alpha was 0.84–0.90 when it was developed (Schwirian, 1978) and 0.951 in the present study. Data Collection This study was approved by the institutional review board of the university in which data collection occurred. Informed consent was exempted because the SBA was considered to be part of regular course development and evaluation. Nonetheless, one of the researchers (not a course instructor) informed the students that the data collected would be used for research purposes. Students were also informed that their grades would not be affected by their decision to either submit data or withhold personal data. SBAs for all students were videotaped; the recordings were subsequently rated by the GRS and checklist. All selfreported data were collected after simulation sessions. Analysis Statistical analysis was performed using SPSS ver.21.0. For descriptive analysis, we used means standard deviations and frequencies with percentages. We examined differences between the asthma and diabetes groups using the t-test and the χ2 test. We also used Pearson and Spearman correlation coefficients to explore associations between competencies. In addition, ANOVA was used to determine if the mean of other nursing competencies differed by groups of GRS and post hoc analysis was conducted using Scheffe's test. Results General characteristics and descriptive statistics of the six competency evaluations of nursing students are presented in Table 1. The students were all women in their early 20s. About half of the students had previous simulation experience, the mean length of which was 1.5– 1.8 h. More students in the asthma group passed the SBA according to the GRS than in the diabetes group (87.3% vs. 64.6%, χ2 = 7.383, p = .007). SBA checklist scores were also higher in the asthma group than in the diabetes group (49.0 ± 15.3 vs. 39.5 ± 13.0, t = 3.370, p = .001). Their average GPA was 3.5–3.6 out of a possible 4.5. The average knowledge score was 8.0 ± 1.4 in the asthma group and 7.9 ± 1.4 in the diabetes group. The average self-efficacy score was 6.8 ± 1.3 in the asthma group and 6.8 ± 1.4 in the diabetes group. The average score on the Six-D Scale was 2.2 ± 0.4 in the asthma group and 2.1 ±
Table 2 Correlations among types of competencies of nursing students (N = 103). SBA
Asthma group SBA Checklist GPA Knowledge Self-efficacy Six-D Scale Diabetes group SBA checklist GPA Knowledge Self-efficacy Six-D Scale
GPA r (p)
Knowledge r (p)
Self-efficacy r (p)
.508 (b .001) .054 (.693) .207 (.129) .352 (.008)
1 .352 (.008) .245 (.071) .091 (.508)
1 .140 (.309) .027 (.843)
1 .244 (.072)
1 .201 (.171) .107 (.470) .339 (.018) .373 (.009)
1 .193 (.190) .066 (.656) .077 (.605)
1 .094 (.526) .152 (.304)
1 .401 (.005)
GRS rho (p)
Checklist r (p)
.763 (b.001) .413 (.002) .022 (.904) .005 (.359) .266 (.049) .475 (.001) .185 (.207) −.111 (.451) .205 (.162) .063 (.671)
SBA = simulation-based assessment; GRS = global rating scale; GPA = grade point average; Six-D Scale = Six-Dimension Scale of Nursing Performance
340
K. Park et al. / Nurse Education Today 36 (2016) 337–341
Table 3 Comparison of nursing student competencies using the global rating scale. SBA - Global Rating Scale, Mean (SD) Clear fail (a) Asthma group Checklist 34.4 (11.4)
F(P) post hoc
Borderline pass (b)
Clear pass (c)
Very good pass (d)
41.6 (10.6)
59.6 (8.9)
75.0 (5.9)
GPA
3.4 (0.4)
3.3 (0.4)
3.7 (0.3)
3.8 (0.3)
Knowledge Self-efficacy Six-D Scale
8.0 (2.2) 6.5 (2.1) 1.9 (0.5)
7.8 (1.3) 6.7 ((1.1) 2.2 (0.4)
8.4 (1.2) 6.8 (1.4)) 2.4 (0.5)
7.5 (1.3) 7.1 (1.5) 2.3 (0.4)
Diabetes group Checklist 31.5 (9.4)
42.6 (12.6)
47.1 (13.9)
56.5*
3.6 (0.4) 7.7 (1.5) 6.8 (1.0) 2.0 (0.4)
3.6 (0.6) 7.8 (1.5) 7.1 (1.3) 2.2 (0.5)
4.0* 8.0* 9.7* 2.6*
GPA Knowledge Self-efficacy Six-D Scale
25.757 (b .001) a, b b c, d 6.095 (.001) a, b b c, d 0.771 (.516) 0.180 (.910) 1.889 (.143)
Table 4 presents the mean scores of five subdomains of the SBA checklists per grades of SBA-GRS in both scenarios. Because SBA-GRS differentiated SBA scores well in both scenarios as presented in Table 3, we explored which of the five subdomains on the SBA checklist was responsible. In general, total SBA scores were highest for physical examination (67.7 ± 23.4 in the asthma group, 70.1 ± 25.0 in the diabetes group) and lowest for assessment by interview (37.9 ± 19.7 in the asthma group, 26.9 ± 20.5 in the diabetes group). In the asthma scenario, scores in all subdomains, except health education, were well differentiated by SBA-GRS. However, in the diabetes scenario, only the intervention domain (F = 4.670, p = .006) was well differentiated. Discussion
3.5 (0.4) 8.1 (1.3) 6.7 (1.3) 2.0 (0.6)
4.790 (.006) a b b, c** 0.630 (.599) 0.350 (.790) 2.247 (.096) 0.609 (.613)
SBA = simulation-based assessment; GPA = grade point average; Six-D Scale = Six Dimension Scale of Nursing Performance. * n = 1. SD was not calculated; ** Scheffe's test was performed on a, b, and c because d had only 1 case.
0.5 in the diabetes group. GPA, knowledge, self-efficacy, and scores on the Six-D Scale were not statistically different between the two groups. Table 2 presents associations between nursing competencies. SBAGRS showed a statistically significant positive association with the SBA checklist in both groups (rho = .763, p b .001 in the asthma group; rho = .475, p = . 001 in the diabetes group) and with GPA (rho = .413, p = .002) and the Six-D Scale only in the asthma group (rho = .266, p = .049). The SBA checklist showed statistically significant positive correlations with GPA (r = .508, p b .001) and scores on the Six-D Scale (r = .352, p = .008) in the asthma group. The SBA checklist was also significantly and positively correlated with self-efficacy (r = .339, p = .018) and scores on the Six-D Scale (r = .373, p = .009) in the diabetes group. Table 3 presents how well SBA-GRS scores differentiated students' achievement assessed with the SBA checklist, GPA, knowledge, selfefficacy, and Six-D Scale scores. SBA-GRS differentiated well scores of SBA checklist in both groups (F = 25.757, p b .001 in the asthma group; F = 4.790, p = .006 in the diabetes group) and GPA in the asthma group (F = 6.095, p b .001). In particular, the SBA-GRS in the asthma scenario differentiated students with a clear pass or a higher grade from those with a borderline pass or a clear fail grade. In contrast, the SBAGRS in the diabetes scenario differentiated students with a borderline pass or higher grade from those with a clear fail grade.
This cross-sectional study describes the development of an SBA to evaluate the clinical competencies of nursing students using a GRS and a checklist and compares the results with other evaluation methods such as GPA, self-efficacy, knowledge, and the Six-D Scale. We found that SBA was feasible and presented reasonable relationships with other evaluation methods. Three important points bear emphasis. First, the GRS and checklist used in SBA showed a high correlation with each other, which is consistent with a previous study (Kim et al., 2009). A recent review study found .76 of correlation in general between GRS and checklist (Ilgen et al., 2015). Furthermore, the GRS used in the SBA was found to differentiate checklist scores well, in particular, to differentiate grades clear fail and what in both scenarios, which is exactly what educators need for student evaluation. However, the distinction was less evident in the diabetes module. The main reason seems to be that only one student obtained “a very good pass” in the diabetes scenario, which was more difficult than the asthma scenario. In view of the GRS and checklist's similar results, both seem to offer unique contributions to the assessment of nursing students in SBA settings. Previous researchers have recommended that the checklist may be useful in assessing education levels on an ongoing basis and in providing progressive feedback to students about their skill levels. In like fashion, the GRS may be useful in final evaluations to determine whether students are ready to move on to a higher level. Second, the correlation between the GRS and checklist and among various types of student competencies differed by scenario type, a reflection perhaps on the different difficulty levels of the two scenarios. More statistically significant correlations were observed in the asthma scenario. This result is consistent with a previous study (Turner et al., 2014) that explored the relationship between GRS and checklist scores in an OSCE with physical therapy students and found less correlation between the two for more difficult OSCE stations. This finding may indicate that a scenario's level of difficulty, due to the demands of complex skills or complicated problem solving strategies, can affect correlations
Table 4 Scores of SBA checklist subdomains using the global rating scale (N = 103). Subdomains of SBA Checklist
Global Rating Scale, Mean (SD) All
Asthma Assessment by interview Physical examination Intervention Health education Professional attitude Diabetes group Assessment by interview Physical examination Intervention Health education Professional attitude
Clear fail (a)
Borderline pass (b)
Clear pass (c)
Very good pass (d)
Mean (SD)
[95% CI]
37.9 (19.7) 67.7 (23.4) 48.6 (25.6) 66.4 (37.3) 42.9 (27.3)
[32.6, 43.2] [61.4, 74.1] [41.7, 55.6] [56.3, 76.5] [35.5, 50.3]
24.5 (10.8) 50.0 (20.4) 35.7 (31.8) 57.1 (34.5 25.7 (22.3
30.2 (15.8) 63.5 (23.7) 38.5 (22.6) 59.6 (40.0 35.4 (29.6
47.6 (18.3) 75.0 (19.2) 63.9 (19.6) 72.2 (35.2 55.6 (18.9
67.9 (7.1) 93.8 (12.5) 68.8 (12.5) 100.0 (0.0) 65.0 (19.1)
26.9 (20.5) 70.1 (25.0) 47.9 (33.6) 38.9 (26.9) 47.1 (24.9)
[21.0, 32.8] [62.9, 77.4] [38.2, 57.7] [31.1, 46.7] [39.8, 54.3]
20.9 (16.6) 68.6 (18.5) 29.4 (33.1) 29.4 (28.6) 38.8 (23.9)
30.6 (22.5) 69.4 (29.4) 58.3 (26.5) 43.1 (23.0) 46.7 (22.6)
24.1 (19.1) 77.8 (27.2) 66.7 (36.5) 38.9 (25.1) 70.0 (27.6)
55.6* 66.7* 0.0* 100.0* 60.0*
F(P)
Post hoc analysis
10.715 (b0.001) 4.630 (.006) 6.355 (.001) 1.722 (.174) 4.453 (.007)
abbbcbd a b b, c b d a b b, c b d
1.473 (.235)** .209 (.890)** 4.670 (.006)** 2.933 (.044)** 2.660 (.060)**
SBA = simulation-based assessment; SD = standard deviation; * n = 1. SD was not calculated;** Scheffe's test was done on a, b, and c because d had only 1 case.
a b b, c b d
a b b, c a=b=c
K. Park et al. / Nurse Education Today 36 (2016) 337–341
between evaluation methods. In other words, the academic level a student has attained in a nursing program can affect correlations between methods. Further studies should explore the effect of scenario difficulty on performance during SBA and correlations between methods with different levels of nursing students. Finally, we found that the GRS, like most nursing interventions that involve skills, was best suited in determining student competency. This result is somewhat consistent with a study (Morgan et al., 2001) that compared the GRS and checklist in an OSCE using an anesthesia simulator for medical students. GRS scores were better correlated with checklist scores for technical skills than other domains. We believe that this is because examiners observe skills traditionally viewed as “real” hospital tasks more carefully than behaviors associated with communication, empathy, or safety. Thus, we suggest that when educators wish to evaluate these behaviors, GRS with more than a single item, which can cover those areas separately, might be appropriate. Limitations This study has several limitations. First, because students were recruited from one university, the generalizability of our findings is limited. Second, the students were not randomly allocated to the treatment groups and scenarios. Nonrandom assignment to the treatment groups can threaten internal validity. In addition, nonrandom assignment to the scenarios may cause measurement bias because checklist scores might be topic-specific and may not appropriately reflect student competency. Third, because one examiner assessed both the SBA-GRS and checklist, it is plausible that scoring the checklist before the GRS may have led to over-correlation between the two methods. However, a previous study (Regehr et al., 1998) reported that GRS scores were not meaningfully affected by knowledge of checklist results. Finally, the checklists were developed by the authors and reviewed by experts in clinical and nursing education. Further research should be conducted to establish better evidence of validity, such as construct validity. Conclusion This study found that SBA was feasible in evaluating the clinical competency of nursing students and the results, using both a GRS and a checklist, presented expected correlations with other students' competencies. We also found that the correlation between the GRS and checklist and among other students' competencies can be affected by the difficulty of scenarios. Furthermore, examiners appeared to focus more on skills than other domains. We recommend that the number of items on a GRS should be adjusted based on the purpose of the examination. We recommend further that research should be conducted in other areas of nursing education such as emergency response, communication, and interprofessional team work, which are difficult to assess in real clinical settings. In particular, a study should be conducted to test the psychometric test of GRS on a variety of topics. This would allow nurse educators the opportunity to evaluate examinees in real clinical environments in a reliable, valid way. References Bandura, A., 1997. The anatomy of stages of changes. Am. J. Health Promot. 12, 1–8. http:// dx.doi.org/10.4278/0890-1171-12.1.8. Burns, T.L., DeBaun, M.R., Boulet, J.R., Murray, G.M., Murray, D.J., Fehr, J.J., 2013. Acute care of pediatric patients with sickle cell disease: a simulation performance assessment. Pediatr. Blood Cancer 60, 1492–1498. http://dx.doi.org/10.1002/pbc.24558.
341
Dong, T., Swygert, K.A., Durning, S.J., Saguil, A., Gilliland, W.R., Cruess, D., ... Artino Jr., A.R., 2014. Validity evidence for medical school OSCEs: associations with USMLE® step assessments. Teach. Learn. Med. 26, 379–386. http://dx.doi.org/10.1080/10401334. 2014.960294. Eftekhar, H., Labaf, A., Anvari, P., Jamali, A., Sheybaee-Moghaddam, F., 2012. Association of the pre-internship objective structured clinical examination in final year medical students with comprehensive written examinations. Med. Educ. Online 17. http://dx.doi. org/10.3402/meo.v17i0.15958. Fehr, J.J., Boulet, J.R., Waldrop, W.B., Snider, R., Brockel, M., Murray, D.J., 2011. Simulationbased assessment of pediatric anesthesia skills. J. Am. Soc. Anesthesiol. 115, 1308–1315. http://dx.doi.org/10.1097/ALN.0b013e318238bcf6. Hayden, J., 2010. Use of simulation in nursing education: national survey results. J. Nurs. Regul. 1, 52–57. http://dx.doi.org/10.1016/S2155-8256(15)30335-5. Henrichs, B.M., Avidan, M.S., Murray, D.J., Boulet, J.R., Kras, J., Krause, B., ... Evers, A.S., 2009. Performance of certified registered nurse anesthetists and anesthesiologists in a simulation-based skills assessment. Anesth. Analg. 108, 255–262. http://dx.doi.org/ 10.1213/ane.0b013e31818e3d58. Hodges, B., McIlroy, J.H., 2003. Analytic global OSCE ratings are sensitive to level of training. Med. Educ. 37, 1012–1016. http://dx.doi.org/10.1046/j.1365-2923.2003.01674.x. Ilgen, J.S., Ma, I.W., Hatala, R., Cook, D.A., 2015. A systematic review of validity evidence for checklists versus global rating scales in simulation‐based assessment. Med. Educ. 49, 161–173. http://dx.doi.org/10.1111/medu.12621. Jaffer, U., Normahani, P., Singh, P., Aslam, M., Standfield, N.J., 2015. The effect of a simulation training package on skill acquisition for duplex arterial stenosis detection. J. Surg. Educ. 72, 310–315. http://dx.doi.org/10.1016/j.jsurg.2014.09.013. Kim, J., Neilipovitz, D., Cardinal, P., Chiu, M., 2009. A comparison of global rating scale and checklist scores in the validation of an evaluation tool to assess performance in the resuscitation of critically ill patients during simulated emergencies (abbreviated as “CRM simulator study IB”). Simul. Healthc. 4, 6–16. http://dx.doi.org/10.1097/SIH. 0b013e3181880472. Klein, C.J., Fowles, E.R., 2009. An investigation of nursing competence and the competency outcomes performance assessment curricular approach: senior students' selfreported perceptions. J. Prof. Nurs. 25, 109–121. http://dx.doi.org/10.1016/j. profnurs.2008.08.006. Liddle, C., 2014. The objective structured clinical examination. Nurs. Times 110, 1–4 (Retrieved from http://www.nursingtimes.net/Journals/2014/08/22/d/l/h/270814-Theobjective-structured-clinical-examination.pdf). McIlroy, J.H., Hodges, B., McNaughton, N., Regehr, G., 2002. The effect of candidates' perceptions of the evaluation method on reliability of checklist and global rating scores in an objective structured clinical examination. Acad. Med. 77, 725–728. http://dx. doi.org/10.1097/00001888-200207000-00018. Mitchell, M.L., Henderson, A., Groves, M., Dalton, M., Nulty, D., 2009. The objective structured clinical examination (OSCE): optimising its value in the undergraduate nursing curriculum. Nurse Educ. Today 29, 398–404. http://dx.doi.org/10.1016/j.nedt.2008. 10.007. Morgan, P.J., Cleave-Hogg, D., Guest, C.B., 2001. A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator. Acad. Med. 76, 1053–1055. http://dx.doi.org/10.1097/00001888-200110000-00016. Palese, A., Bulfone, G., Venturato, E., Urli, N., Bulfone, T., Zanini, A., ... Dante, A., 2012. The cost of the objective structured clinical examination on an Italian nursing bachelor's degree course. Nurse Educ. Today 32, 422–426. http://dx.doi.org/10.1016/j.nedt. 2011.03.003. Park, K.O., Ahn, Y.M., Kang, N.R., Lee, M.J., Sohn, M., 2013. Psychometric evaluation of a six dimension scale of nursing performance and student nurse stress index using an objective structured clinical examination-modules for asthma and type 1 diabetes. Child Health Nurs. Res. 19, 85–93. http://dx.doi.org/10.4094/chnr.2013.19.2.85. Regehr, G., MacRae, H., Reznick, R.K., Szalay, D., 1998. Comparing the psychometric properties of checklists and global ratings scales for assessing performance on an OSCEformat examination. Acad. Med. 73, 993–997. http://dx.doi.org/10.1097/00001888199809000-00020. Schoenmakers, B., Wens, J., 2014. The objective structured clinical examination revisited for postgraduate trainees in general practice. Int. J. Med. Educ. 5, 45–50. http://dx. doi.org/10.5116/ijme.52eb.f882. Schwirian, P.M., 1978. Evaluating the performance of nurses: a multidimensional approach. Nurs. Res. 27, 347–350. Turner, K., Bell, M., Bays, L., Lau, C., Lai, C., Kendzerska, T., ... Davies, R., 2014. Correlation between global rating scale and specific checklist scores for professional behaviour of physical therapy students in practical examinations. Educ. Res. Int. 2014, 1–6. http://dx.doi.org/10.1155/2014/219512.
Contents lists available at ScienceDirect
Nurse Education Today journal homepage: www.elsevier.com/nedt
Development of a simulation-based assessment to evaluate the clinical competencies of Korean nursing students☆ Kyongok Park a, Youngmee Ahn b, Narae Kang b, Min Sohn b,⁎ a b
Department of Nursing, Far East University, Eumseong, Republic of Korea Department of Nursing, Inha University, Incheon, Republic of Korea
a r t i c l e
i n f o
Article history: Accepted 21 August 2015 Keywords: Asthma Diabetes Global rating scale Nursing students Simulation
s u m m a r y Objectives: To describe a simulation-based assessment (SBA) to evaluate the clinical competencies of nursing students in children's health and to compare its results with grade point average (GPA), self-efficacy, topic-specific knowledge, and self-reported clinical competency using the Six-D Scale. Methods: This cross-sectional, descriptive study recruited nursing students from a children's health clinical practicum. Students were assigned to either an asthma (n = 55) or a type 1 diabetes (n = 48) care scenario conducted on a high-fidelity simulator. Clinical competencies were assessed using the global rating scale (GRS) and a checklist. Results: Data on 103 students were analyzed. The SBA-GRS indicated that 64.6%–87.3% of students passed. The SBA-GRS showed a statistically significant positive association with the SBA checklist in both the asthma (rho = .763, p b .001) and the type 1 diabetes (rho = .475, p = .001) group. In the asthma group, the SBA-GRS and checklist showed statistically significant associations with GPA (rho = .413, p = .002 vs. r = .508, p b .001) and the Six-D Scale (rho = .266, p = .049 vs. r = .352, p = .008); in the diabetes group, only the SBA checklist showed a statistically significant association with self-efficacy (r = .339, p = .018) and the Six-D Scale (r = .373, p = .009). Four groups by SBA-GRS had statistically significant differences in scores on the SBA checklist in both groups (F = 25.757, p b .001 in the asthma group; F = 4.790, p = .006 in the diabetes group) and GPA only in the asthma groups (F = 6.095, p b .001). Conclusion: SBA was found to be feasible for nursing students. The GRS and checklist were reasonably correlated with other evaluation methods of student competency, but correlations were better with easier scenarios. © 2015 Elsevier Ltd. All rights reserved.
Introduction The assessment of clinical competency has always been a key component of clinical education for health professionals. In addition to traditional methods such as written tests and clinical observations, direct evaluation methods conducted in standard environments have attracted the attention of nursing educators. For example, objective structured clinical examination (OSCE) has been widely used in many health professions programs, including nursing. OSCE has been shown to be reliable and valid in assessing medical students (Dong et al., 2014), nursing students (Mitchell et al., 2009), and postgraduate professionals (Schoenmakers & Wens, 2014). However, adapting OSCE can be
☆ This study was supported by the Basic Science Research Program of the National Research Foundation of Korea and funded by the Ministry of Education, Science, and Technology (2011-0009627). ⁎ Corresponding author at: Department of Nursing, Inha University, 100 Inharo, NamGu, Incheon, South Korea. Tel.: +82 32 860 8212; fax: +82 32 874 5880. E-mail address: [email protected] (M. Sohn).
http://dx.doi.org/10.1016/j.nedt.2015.08.020 0260-6917/© 2015 Elsevier Ltd. All rights reserved.
problematic, particularly when resources are limited. OSCE requires multiple clinical scenarios and space, equipment, supplies, standardized patients, and trained examiners to evaluate multiple observations of individual students (McIlroy et al., 2002). Because these requirements are so expensive (Palese et al., 2012), OSCE is not available in many Korean nursing programs. Nonetheless, standardized objective assessment of clinical competency is still an important part of nursing education, and simulationbased assessment (SBA) could be a useful alternative. Not only has simulation been widely used for the clinical education of nursing students and nurses but it has also been used to assess the clinical competencies of surgeons (Jaffer et al., 2015), medical residents (Burns et al., 2013; Fehr et al., 2011), and nurse anesthetists (Henrichs et al., 2009). Compared with OSCE, SBA also has unique benefits for student evaluation: its operating costs are smaller, it is less demanding of examiners, and it provides students with almost unlimited opportunities to practice. The most commonly used evaluation tool in OSCE or SBA of students is the checklist. A checklist consists of a series of items that are usually rated dichotomously: 1 (“performed”) or 0 (“not performed”). It
338
K. Park et al. / Nurse Education Today 36 (2016) 337–341
facilitates a thorough assessment of inexperienced examinees at every stage of performance. In addition, educators often use checklist results to advise students how they performed on individual items. However, checklists are not always the best option. They are case-specific evaluation tools, and their items are scenario-dependent, which is highly demanding for both developers and examiners. Furthermore, previous studies have concluded that checklists make it difficult to evaluate higher levels of performance such as integrity, prioritization, and efficiency (Hodges & McIlroy, 2003). The global rating scale (GRS) can also be used to assess clinical competency. The GRS typically requires examiners to rate the overall performance of examinees based on expert judgment and often uses only a few items, on occasions even a single item (Hodges & McIlroy, 2003). Global rating scales have several benefits over checklists: They are easily scored, allow greater flexibility in judgment, take less time to administer, and provide a better means of assessing improvements, from the novice to the expert level (Kim et al., 2009). In addition, the GRS seems to capture aspects (e.g., empathy, communication, and short and effective skills) that are not well-addressed in textbooks but are usually found in clinical experts. Moreover, a well-developed GRS affords examiners the opportunity to test examinees in real clinical settings. In SBA, either the GRS or checklist can be used to assess student competencies. Ilgen et al. (2015) recently reported that both methods had good reliability and validity when used with SBA (Ilgen et al., 2015). However, the evidence is still limited, particularly in nursing education. The association between these two methods and other types of assessment of student competencies during nursing programs is lacking. For example, grade point average (GPA), self-efficacy, knowledge of the target topic, and self-reported competency, as determined by the Six-Dimension Scale of Nursing Performance (Six-D Scale), have been used in previous studies of OSCE of students (Eftekhar et al., 2012; Park et al., 2013), but these have rarely been investigated in conjunction with SBA. This study, therefore, was undertaken to develop an SBA for nursing students and to compare the efficacy of a GRS and a checklist to measure student competency with other methods of evaluation. We were particularly interested in children's health. The clinical practicum of nursing education for children has been made difficult by a decreasing number of clients and clinical placements. This means that students have fewer opportunities to practice nursing care, and educators are more pressured to ensure that students can perform well in real care situations. For these reasons, children's health is one area where simulation is widely used for the clinical education of nursing students (Hayden, 2010). For this study, we selected asthma and type 1 diabetes, two chronic health conditions that are common in children. Our specific aims were (1) to describe the development of an SBA using a GRS and checklist to assess the clinical competencies of nursing students for children with asthma or diabetes and (2) to compare SBA results with different types of competency tests such as GPA, self-efficacy, knowledge of the topics, and the Six-D Scale. Methods Study Design and Participants This was a descriptive, cross-sectional study. Data were collected from March 2012 to June 2013. The study participants were conveniently recruited from a group of fourth year nursing students who were participating in a clinical practicum on children's health in a university located in Incheon, South Korea. Development of SBA Modules for Asthma and Type 1 Diabetes Two SBA modules, one for asthma and the other for type 1 diabetes, were developed by a focus group, which included two nursing faculty
and two nursing researchers in children's health and nursing simulation education. After the modules had been drafted, they were reviewed by clinicians (i.e., a pediatrician, a clinical nurse specialist, and a nurse manager in a pediatric department) and subsequently revised based on their feedback. Scenarios on asthma and diabetes were developed with two levels of difficulty to evaluate their effect on study outcomes. The principals of both scenarios were young teenagers. The asthma scenario involved a teenager who visited an emergency department with shortness of breath; the diabetes scenario involved an inpatient with a cold who was admitted for high blood sugar. The diabetes scenario was the more difficult scenario because the module presented at least two nursing problems (i.e., high blood sugar and dehydration), while the asthma scenario involved only an acute exacerbation of asthma. Both modules comprised a planning phase (15 min), a simulation performance phase (15 min), and a documentation/survey completion phase (15 min). Each student performed one of two scenarios on the first day of the 5-day practicum. The scenario was given alternatively among the two scenarios per week. Students were provided with written information that included a summary of each scenario, the physician's order, medication administration records, nursing records, drug information, and a list of actions that the students were required to perform. These actions included assessment, interventions, health promotion counseling, and presentation of an appropriate professional attitude. After students had finished planning, they proceeded to the simulation phase. While an operator controlled the simulator, an examiner observed the students through a one-way mirror and recorded their performances.
Measurements We collected data on basic characteristics and six measures of competencies. The basic characteristics consisted of age, gender, experience of simulation, and amount of previous simulation use. The six measures of competencies were SBA-GRS and checklist scores, GPA, knowledge about asthma or diabetes, self-efficacy, and Six-D Scale scores. GPA, the average grade of six semesters, was obtained from the department of nursing. The SBA-GRS and checklist, questionnaires on asthma and diabetes knowledge, and self-efficacy were developed by the focus group and reviewed by clinicians, as described above.
SBA-GRS Researchers may use 3- to 5-point Likert scales in a GRS (Liddle, 2014). We selected a 4-point Likert scale to avoid examiners selecting, perhaps unconsciously, a middle grade. Based on their performance, students received one of four grades: “very good pass,” “clear pass,” “borderline pass,” and “fail.”
SBA Checklist The asthma scenario checklist comprised five domains (22 items): assessment by interview (7 items), physical examination (4 items), intervention (4 items), health education (2 items), and professional attitude (5 items). Each item was coded either 1 (“correctly performed”) or 0 (“not correctly performed”). Domain scores were expressed as percentages of total possible scores. Total checklist scores were defined as the average scores of the five domains, which meant that domains were equally weighted. The diabetes scenario checklist also consisted of five domains, but it had 23 items: assessment by interview (9 items), physical examination (3 items), intervention (3 items), health education (3 items), and professional attitude (5 items). The same scoring system was applied to this scenario. Inter-rater reliability of the checklists was 0.75 for the asthma scenario and 0.87 for the diabetes scenario.
K. Park et al. / Nurse Education Today 36 (2016) 337–341 Table 1 Characteristics of nursing students by scenario (N = 103). Student groups Asthma (n = 55)
t-test or χ2 (p)
Diabetes (n = 48)
Frequency (%) or mean (SD) Age (year) 22.0 (1.1) 21.5 (0.7) 2.74 (.007) Experience with simulation 29 (52.7) 26 (54.2) 0.021 (N.999) Length of simulation experience 1.5 (0.9) 1.8 (1.2) −1.278 (.208) among experienced students (hours) SBA-GRS, Pass 48 (87.3) 31 (64.6) 7.383 (.007) Very good pass 4 (7.3) 1 (2.1) Clear pass 18 (32.7) 6 (12.5) Borderline pass 26 (47.3) 24 (50.0) Clear fail 7 (12.7) 17 (35.4) SBA Checklist 49.0 (15.3) 39.5 (13.0) 3.370 (.001) GPA 3.5 (0.4) 3.6 (0.4) −1.227 (.223) Knowledge 8.0 (1.4) 7.9 (1.4) .530 (.597) Self-efficacy 6.8 (1.3) 6.8 (1.2) −.305 (.761) Six-D Scale 2.2 (0.4) 2.1 (0.5) 1.987 (.050) SD = standard deviation; SBA = simulation-based assessment; GRS = global rating scale; GPA = grade point average; Six-D Scale = Six-Dimension Scale of Nursing Performance.
Knowledge Knowledge about asthma and diabetes was assessed using a 10-item quiz, which included questions on assessment, intervention, medication, and health promotion for children with asthma or type 1 diabetes. Scores for total knowledge were based on the number of items students answered correctly. Self-efficacy Based on Bandura’s (1997) social cognitive theory, we developed the instrument to examine how confident students were in providing nursing care to children with asthma or diabetes. The scope of that care involved assessment, prioritizing nursing problems, intervention, and health promotion education. Self-efficacy in nursing care for children with asthma or diabetes included 30 and 32 items, respectively. Students graded their confidence levels from 0 (“not at all confident”) to 10 (“absolutely confident”). Total scores were defined as the mean of all items and ranged from 0 to 10. Higher scores indicated higher levels of confidence. Cronbach's alpha was 0.945 for asthma and 0.959 for diabetes. The Six-D Scale Originally developed to evaluate the competency of nurses by assessing their level of confidence for selected nursing actions (Schwirian, 1978), the Six-D Scale has also been used for nursing students (Klein & Fowles, 2009). The scale comprises six subscales
339
(52 items): leadership (5 items), critical care (7 items), teaching and collaboration (11 items), planning and evaluation (7 items), and interpersonal relations and communication (12 items). Students were asked to rate their competencies from 1 (“not very well”) to 4 (“very well”). Total scores were defined as the mean item score; higher scores indicated greater nursing competency. We used a Korean version of the Six-D Scale, which was developed and tested on Korean nursing students (Park et al., 2013). Cronbach's alpha was 0.84–0.90 when it was developed (Schwirian, 1978) and 0.951 in the present study. Data Collection This study was approved by the institutional review board of the university in which data collection occurred. Informed consent was exempted because the SBA was considered to be part of regular course development and evaluation. Nonetheless, one of the researchers (not a course instructor) informed the students that the data collected would be used for research purposes. Students were also informed that their grades would not be affected by their decision to either submit data or withhold personal data. SBAs for all students were videotaped; the recordings were subsequently rated by the GRS and checklist. All selfreported data were collected after simulation sessions. Analysis Statistical analysis was performed using SPSS ver.21.0. For descriptive analysis, we used means standard deviations and frequencies with percentages. We examined differences between the asthma and diabetes groups using the t-test and the χ2 test. We also used Pearson and Spearman correlation coefficients to explore associations between competencies. In addition, ANOVA was used to determine if the mean of other nursing competencies differed by groups of GRS and post hoc analysis was conducted using Scheffe's test. Results General characteristics and descriptive statistics of the six competency evaluations of nursing students are presented in Table 1. The students were all women in their early 20s. About half of the students had previous simulation experience, the mean length of which was 1.5– 1.8 h. More students in the asthma group passed the SBA according to the GRS than in the diabetes group (87.3% vs. 64.6%, χ2 = 7.383, p = .007). SBA checklist scores were also higher in the asthma group than in the diabetes group (49.0 ± 15.3 vs. 39.5 ± 13.0, t = 3.370, p = .001). Their average GPA was 3.5–3.6 out of a possible 4.5. The average knowledge score was 8.0 ± 1.4 in the asthma group and 7.9 ± 1.4 in the diabetes group. The average self-efficacy score was 6.8 ± 1.3 in the asthma group and 6.8 ± 1.4 in the diabetes group. The average score on the Six-D Scale was 2.2 ± 0.4 in the asthma group and 2.1 ±
Table 2 Correlations among types of competencies of nursing students (N = 103). SBA
Asthma group SBA Checklist GPA Knowledge Self-efficacy Six-D Scale Diabetes group SBA checklist GPA Knowledge Self-efficacy Six-D Scale
GPA r (p)
Knowledge r (p)
Self-efficacy r (p)
.508 (b .001) .054 (.693) .207 (.129) .352 (.008)
1 .352 (.008) .245 (.071) .091 (.508)
1 .140 (.309) .027 (.843)
1 .244 (.072)
1 .201 (.171) .107 (.470) .339 (.018) .373 (.009)
1 .193 (.190) .066 (.656) .077 (.605)
1 .094 (.526) .152 (.304)
1 .401 (.005)
GRS rho (p)
Checklist r (p)
.763 (b.001) .413 (.002) .022 (.904) .005 (.359) .266 (.049) .475 (.001) .185 (.207) −.111 (.451) .205 (.162) .063 (.671)
SBA = simulation-based assessment; GRS = global rating scale; GPA = grade point average; Six-D Scale = Six-Dimension Scale of Nursing Performance
340
K. Park et al. / Nurse Education Today 36 (2016) 337–341
Table 3 Comparison of nursing student competencies using the global rating scale. SBA - Global Rating Scale, Mean (SD) Clear fail (a) Asthma group Checklist 34.4 (11.4)
F(P) post hoc
Borderline pass (b)
Clear pass (c)
Very good pass (d)
41.6 (10.6)
59.6 (8.9)
75.0 (5.9)
GPA
3.4 (0.4)
3.3 (0.4)
3.7 (0.3)
3.8 (0.3)
Knowledge Self-efficacy Six-D Scale
8.0 (2.2) 6.5 (2.1) 1.9 (0.5)
7.8 (1.3) 6.7 ((1.1) 2.2 (0.4)
8.4 (1.2) 6.8 (1.4)) 2.4 (0.5)
7.5 (1.3) 7.1 (1.5) 2.3 (0.4)
Diabetes group Checklist 31.5 (9.4)
42.6 (12.6)
47.1 (13.9)
56.5*
3.6 (0.4) 7.7 (1.5) 6.8 (1.0) 2.0 (0.4)
3.6 (0.6) 7.8 (1.5) 7.1 (1.3) 2.2 (0.5)
4.0* 8.0* 9.7* 2.6*
GPA Knowledge Self-efficacy Six-D Scale
25.757 (b .001) a, b b c, d 6.095 (.001) a, b b c, d 0.771 (.516) 0.180 (.910) 1.889 (.143)
Table 4 presents the mean scores of five subdomains of the SBA checklists per grades of SBA-GRS in both scenarios. Because SBA-GRS differentiated SBA scores well in both scenarios as presented in Table 3, we explored which of the five subdomains on the SBA checklist was responsible. In general, total SBA scores were highest for physical examination (67.7 ± 23.4 in the asthma group, 70.1 ± 25.0 in the diabetes group) and lowest for assessment by interview (37.9 ± 19.7 in the asthma group, 26.9 ± 20.5 in the diabetes group). In the asthma scenario, scores in all subdomains, except health education, were well differentiated by SBA-GRS. However, in the diabetes scenario, only the intervention domain (F = 4.670, p = .006) was well differentiated. Discussion
3.5 (0.4) 8.1 (1.3) 6.7 (1.3) 2.0 (0.6)
4.790 (.006) a b b, c** 0.630 (.599) 0.350 (.790) 2.247 (.096) 0.609 (.613)
SBA = simulation-based assessment; GPA = grade point average; Six-D Scale = Six Dimension Scale of Nursing Performance. * n = 1. SD was not calculated; ** Scheffe's test was performed on a, b, and c because d had only 1 case.
0.5 in the diabetes group. GPA, knowledge, self-efficacy, and scores on the Six-D Scale were not statistically different between the two groups. Table 2 presents associations between nursing competencies. SBAGRS showed a statistically significant positive association with the SBA checklist in both groups (rho = .763, p b .001 in the asthma group; rho = .475, p = . 001 in the diabetes group) and with GPA (rho = .413, p = .002) and the Six-D Scale only in the asthma group (rho = .266, p = .049). The SBA checklist showed statistically significant positive correlations with GPA (r = .508, p b .001) and scores on the Six-D Scale (r = .352, p = .008) in the asthma group. The SBA checklist was also significantly and positively correlated with self-efficacy (r = .339, p = .018) and scores on the Six-D Scale (r = .373, p = .009) in the diabetes group. Table 3 presents how well SBA-GRS scores differentiated students' achievement assessed with the SBA checklist, GPA, knowledge, selfefficacy, and Six-D Scale scores. SBA-GRS differentiated well scores of SBA checklist in both groups (F = 25.757, p b .001 in the asthma group; F = 4.790, p = .006 in the diabetes group) and GPA in the asthma group (F = 6.095, p b .001). In particular, the SBA-GRS in the asthma scenario differentiated students with a clear pass or a higher grade from those with a borderline pass or a clear fail grade. In contrast, the SBAGRS in the diabetes scenario differentiated students with a borderline pass or higher grade from those with a clear fail grade.
This cross-sectional study describes the development of an SBA to evaluate the clinical competencies of nursing students using a GRS and a checklist and compares the results with other evaluation methods such as GPA, self-efficacy, knowledge, and the Six-D Scale. We found that SBA was feasible and presented reasonable relationships with other evaluation methods. Three important points bear emphasis. First, the GRS and checklist used in SBA showed a high correlation with each other, which is consistent with a previous study (Kim et al., 2009). A recent review study found .76 of correlation in general between GRS and checklist (Ilgen et al., 2015). Furthermore, the GRS used in the SBA was found to differentiate checklist scores well, in particular, to differentiate grades clear fail and what in both scenarios, which is exactly what educators need for student evaluation. However, the distinction was less evident in the diabetes module. The main reason seems to be that only one student obtained “a very good pass” in the diabetes scenario, which was more difficult than the asthma scenario. In view of the GRS and checklist's similar results, both seem to offer unique contributions to the assessment of nursing students in SBA settings. Previous researchers have recommended that the checklist may be useful in assessing education levels on an ongoing basis and in providing progressive feedback to students about their skill levels. In like fashion, the GRS may be useful in final evaluations to determine whether students are ready to move on to a higher level. Second, the correlation between the GRS and checklist and among various types of student competencies differed by scenario type, a reflection perhaps on the different difficulty levels of the two scenarios. More statistically significant correlations were observed in the asthma scenario. This result is consistent with a previous study (Turner et al., 2014) that explored the relationship between GRS and checklist scores in an OSCE with physical therapy students and found less correlation between the two for more difficult OSCE stations. This finding may indicate that a scenario's level of difficulty, due to the demands of complex skills or complicated problem solving strategies, can affect correlations
Table 4 Scores of SBA checklist subdomains using the global rating scale (N = 103). Subdomains of SBA Checklist
Global Rating Scale, Mean (SD) All
Asthma Assessment by interview Physical examination Intervention Health education Professional attitude Diabetes group Assessment by interview Physical examination Intervention Health education Professional attitude
Clear fail (a)
Borderline pass (b)
Clear pass (c)
Very good pass (d)
Mean (SD)
[95% CI]
37.9 (19.7) 67.7 (23.4) 48.6 (25.6) 66.4 (37.3) 42.9 (27.3)
[32.6, 43.2] [61.4, 74.1] [41.7, 55.6] [56.3, 76.5] [35.5, 50.3]
24.5 (10.8) 50.0 (20.4) 35.7 (31.8) 57.1 (34.5 25.7 (22.3
30.2 (15.8) 63.5 (23.7) 38.5 (22.6) 59.6 (40.0 35.4 (29.6
47.6 (18.3) 75.0 (19.2) 63.9 (19.6) 72.2 (35.2 55.6 (18.9
67.9 (7.1) 93.8 (12.5) 68.8 (12.5) 100.0 (0.0) 65.0 (19.1)
26.9 (20.5) 70.1 (25.0) 47.9 (33.6) 38.9 (26.9) 47.1 (24.9)
[21.0, 32.8] [62.9, 77.4] [38.2, 57.7] [31.1, 46.7] [39.8, 54.3]
20.9 (16.6) 68.6 (18.5) 29.4 (33.1) 29.4 (28.6) 38.8 (23.9)
30.6 (22.5) 69.4 (29.4) 58.3 (26.5) 43.1 (23.0) 46.7 (22.6)
24.1 (19.1) 77.8 (27.2) 66.7 (36.5) 38.9 (25.1) 70.0 (27.6)
55.6* 66.7* 0.0* 100.0* 60.0*
F(P)
Post hoc analysis
10.715 (b0.001) 4.630 (.006) 6.355 (.001) 1.722 (.174) 4.453 (.007)
abbbcbd a b b, c b d a b b, c b d
1.473 (.235)** .209 (.890)** 4.670 (.006)** 2.933 (.044)** 2.660 (.060)**
SBA = simulation-based assessment; SD = standard deviation; * n = 1. SD was not calculated;** Scheffe's test was done on a, b, and c because d had only 1 case.
a b b, c b d
a b b, c a=b=c
K. Park et al. / Nurse Education Today 36 (2016) 337–341
between evaluation methods. In other words, the academic level a student has attained in a nursing program can affect correlations between methods. Further studies should explore the effect of scenario difficulty on performance during SBA and correlations between methods with different levels of nursing students. Finally, we found that the GRS, like most nursing interventions that involve skills, was best suited in determining student competency. This result is somewhat consistent with a study (Morgan et al., 2001) that compared the GRS and checklist in an OSCE using an anesthesia simulator for medical students. GRS scores were better correlated with checklist scores for technical skills than other domains. We believe that this is because examiners observe skills traditionally viewed as “real” hospital tasks more carefully than behaviors associated with communication, empathy, or safety. Thus, we suggest that when educators wish to evaluate these behaviors, GRS with more than a single item, which can cover those areas separately, might be appropriate. Limitations This study has several limitations. First, because students were recruited from one university, the generalizability of our findings is limited. Second, the students were not randomly allocated to the treatment groups and scenarios. Nonrandom assignment to the treatment groups can threaten internal validity. In addition, nonrandom assignment to the scenarios may cause measurement bias because checklist scores might be topic-specific and may not appropriately reflect student competency. Third, because one examiner assessed both the SBA-GRS and checklist, it is plausible that scoring the checklist before the GRS may have led to over-correlation between the two methods. However, a previous study (Regehr et al., 1998) reported that GRS scores were not meaningfully affected by knowledge of checklist results. Finally, the checklists were developed by the authors and reviewed by experts in clinical and nursing education. Further research should be conducted to establish better evidence of validity, such as construct validity. Conclusion This study found that SBA was feasible in evaluating the clinical competency of nursing students and the results, using both a GRS and a checklist, presented expected correlations with other students' competencies. We also found that the correlation between the GRS and checklist and among other students' competencies can be affected by the difficulty of scenarios. Furthermore, examiners appeared to focus more on skills than other domains. We recommend that the number of items on a GRS should be adjusted based on the purpose of the examination. We recommend further that research should be conducted in other areas of nursing education such as emergency response, communication, and interprofessional team work, which are difficult to assess in real clinical settings. In particular, a study should be conducted to test the psychometric test of GRS on a variety of topics. This would allow nurse educators the opportunity to evaluate examinees in real clinical environments in a reliable, valid way. References Bandura, A., 1997. The anatomy of stages of changes. Am. J. Health Promot. 12, 1–8. http:// dx.doi.org/10.4278/0890-1171-12.1.8. Burns, T.L., DeBaun, M.R., Boulet, J.R., Murray, G.M., Murray, D.J., Fehr, J.J., 2013. Acute care of pediatric patients with sickle cell disease: a simulation performance assessment. Pediatr. Blood Cancer 60, 1492–1498. http://dx.doi.org/10.1002/pbc.24558.
341
Dong, T., Swygert, K.A., Durning, S.J., Saguil, A., Gilliland, W.R., Cruess, D., ... Artino Jr., A.R., 2014. Validity evidence for medical school OSCEs: associations with USMLE® step assessments. Teach. Learn. Med. 26, 379–386. http://dx.doi.org/10.1080/10401334. 2014.960294. Eftekhar, H., Labaf, A., Anvari, P., Jamali, A., Sheybaee-Moghaddam, F., 2012. Association of the pre-internship objective structured clinical examination in final year medical students with comprehensive written examinations. Med. Educ. Online 17. http://dx.doi. org/10.3402/meo.v17i0.15958. Fehr, J.J., Boulet, J.R., Waldrop, W.B., Snider, R., Brockel, M., Murray, D.J., 2011. Simulationbased assessment of pediatric anesthesia skills. J. Am. Soc. Anesthesiol. 115, 1308–1315. http://dx.doi.org/10.1097/ALN.0b013e318238bcf6. Hayden, J., 2010. Use of simulation in nursing education: national survey results. J. Nurs. Regul. 1, 52–57. http://dx.doi.org/10.1016/S2155-8256(15)30335-5. Henrichs, B.M., Avidan, M.S., Murray, D.J., Boulet, J.R., Kras, J., Krause, B., ... Evers, A.S., 2009. Performance of certified registered nurse anesthetists and anesthesiologists in a simulation-based skills assessment. Anesth. Analg. 108, 255–262. http://dx.doi.org/ 10.1213/ane.0b013e31818e3d58. Hodges, B., McIlroy, J.H., 2003. Analytic global OSCE ratings are sensitive to level of training. Med. Educ. 37, 1012–1016. http://dx.doi.org/10.1046/j.1365-2923.2003.01674.x. Ilgen, J.S., Ma, I.W., Hatala, R., Cook, D.A., 2015. A systematic review of validity evidence for checklists versus global rating scales in simulation‐based assessment. Med. Educ. 49, 161–173. http://dx.doi.org/10.1111/medu.12621. Jaffer, U., Normahani, P., Singh, P., Aslam, M., Standfield, N.J., 2015. The effect of a simulation training package on skill acquisition for duplex arterial stenosis detection. J. Surg. Educ. 72, 310–315. http://dx.doi.org/10.1016/j.jsurg.2014.09.013. Kim, J., Neilipovitz, D., Cardinal, P., Chiu, M., 2009. A comparison of global rating scale and checklist scores in the validation of an evaluation tool to assess performance in the resuscitation of critically ill patients during simulated emergencies (abbreviated as “CRM simulator study IB”). Simul. Healthc. 4, 6–16. http://dx.doi.org/10.1097/SIH. 0b013e3181880472. Klein, C.J., Fowles, E.R., 2009. An investigation of nursing competence and the competency outcomes performance assessment curricular approach: senior students' selfreported perceptions. J. Prof. Nurs. 25, 109–121. http://dx.doi.org/10.1016/j. profnurs.2008.08.006. Liddle, C., 2014. The objective structured clinical examination. Nurs. Times 110, 1–4 (Retrieved from http://www.nursingtimes.net/Journals/2014/08/22/d/l/h/270814-Theobjective-structured-clinical-examination.pdf). McIlroy, J.H., Hodges, B., McNaughton, N., Regehr, G., 2002. The effect of candidates' perceptions of the evaluation method on reliability of checklist and global rating scores in an objective structured clinical examination. Acad. Med. 77, 725–728. http://dx. doi.org/10.1097/00001888-200207000-00018. Mitchell, M.L., Henderson, A., Groves, M., Dalton, M., Nulty, D., 2009. The objective structured clinical examination (OSCE): optimising its value in the undergraduate nursing curriculum. Nurse Educ. Today 29, 398–404. http://dx.doi.org/10.1016/j.nedt.2008. 10.007. Morgan, P.J., Cleave-Hogg, D., Guest, C.B., 2001. A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator. Acad. Med. 76, 1053–1055. http://dx.doi.org/10.1097/00001888-200110000-00016. Palese, A., Bulfone, G., Venturato, E., Urli, N., Bulfone, T., Zanini, A., ... Dante, A., 2012. The cost of the objective structured clinical examination on an Italian nursing bachelor's degree course. Nurse Educ. Today 32, 422–426. http://dx.doi.org/10.1016/j.nedt. 2011.03.003. Park, K.O., Ahn, Y.M., Kang, N.R., Lee, M.J., Sohn, M., 2013. Psychometric evaluation of a six dimension scale of nursing performance and student nurse stress index using an objective structured clinical examination-modules for asthma and type 1 diabetes. Child Health Nurs. Res. 19, 85–93. http://dx.doi.org/10.4094/chnr.2013.19.2.85. Regehr, G., MacRae, H., Reznick, R.K., Szalay, D., 1998. Comparing the psychometric properties of checklists and global ratings scales for assessing performance on an OSCEformat examination. Acad. Med. 73, 993–997. http://dx.doi.org/10.1097/00001888199809000-00020. Schoenmakers, B., Wens, J., 2014. The objective structured clinical examination revisited for postgraduate trainees in general practice. Int. J. Med. Educ. 5, 45–50. http://dx. doi.org/10.5116/ijme.52eb.f882. Schwirian, P.M., 1978. Evaluating the performance of nurses: a multidimensional approach. Nurs. Res. 27, 347–350. Turner, K., Bell, M., Bays, L., Lau, C., Lai, C., Kendzerska, T., ... Davies, R., 2014. Correlation between global rating scale and specific checklist scores for professional behaviour of physical therapy students in practical examinations. Educ. Res. Int. 2014, 1–6. http://dx.doi.org/10.1155/2014/219512.