- Email: [email protected]
Influential Factors of Student Evaluations of Teaching in a Nursing Program
Teaching and Learning in Nursing 13 (2018) 86–94
Contents lists available at ScienceDirect
Teaching and Learning in Nursing journal homepage: www.jtln.org
Influential Factors of Student Evaluations of Teaching in a Nursing Program Jeong IL Cho a,⁎, Kawther Hamash b, Koichiro Otani c, Nila Reimer b, Yihao Deng d a
Department of Professional Studies, Purdue University Fort Wayne, 468, USA Department of Nursing, Indiana University Fort Wayne, 468, USA Department of Public Policy, Purdue University Fort Wayne, 468, USA d Department of Mathematical Sciences, Purdue University Fort Wayne, 468, USA b c
a r t i c l e
i n f o
Article history: Accepted 6 January 2018 Available online xxxx Keywords: Student evaluation of teaching Teaching excellence Nursing students Motivation SET
a b s t r a c t We used a logistic regression model to analyze data from routinely collected student evaluation of teaching from 2,211 nursing students over 8 semesters. The results showed that not all aspects of teaching are equally influential in their effects on students' overall rating of instructors' teaching as excellent. Influential aspects include motivation, individual assistance, presentation of concepts, environment, and practical application. The most influential evaluation aspect is motivation. © 2017 Published by Elsevier Inc. on behalf of Organization for Associate Degree Nursing.
Introduction Nursing educators have voiced a significant issue with the student evaluation of teaching (SET) and called for more systematic studies on the effective use of SET in nursing education programs (Annan, Tratnack, Rubenstein, Metzler-Sawin, & Hulton, 2013). Valiga (2017) called for action in achieving and sustaining excellence in teaching in nursing through improving the pedagogical expertise of faculty and using science in teaching and learning to design specific learning experiences to benefit nursing students for the complex and challenging future of nursing. More research should be directed to identifying nursing students' needs and exploring the teaching effectiveness of nursing faculty (Lovric et al., 2014). Even though there have been debates on the usability, reliability, and validity of SET in nursing education (Oermann, 2017), it is perceived as a critical and logical tool faculty and administrators can use to understand students' constantly changing needs and evaluation of faculty's instruction and teaching effectiveness (Agbetsiafa, 2010). Findings about how students evaluate the teaching effectiveness of faculty as excellently drawn from SET have a potential to be informative and practical in departmental- and program-level discussions among faculty as
⁎ Corresponding author. Tel.: +1 260 481 6454; fax: +1 260 481 5408. E-mail address: [email protected] (J. Cho).
part of the professional development activities to achieve teaching excellence in evolving nursing education. We investigated the influence of multidimensional evaluation aspects of SET on the overall rating as excellent with the intention of providing insights about the quality of nursing instructors' instructional practices perceived by students and the significant impact of certain instructional activities on students' overall evaluation of teaching excellence of their instructors. The systematic approach of the current study in the analysis of the routinely collected SET data at a midwestern state university responds to the continuing concern of faculty about how useful SET can be for the evaluation of their teaching effectiveness (Al-Maamari, 2015) and highlights the qualities that students value in excellent nursing instructors. Literature Review Student Evaluations of Teaching in Higher Education Institutions Student evaluations of teaching (SET) in higher education have received increasing administrative, instructional, and research attention because of the significant impact they have on judgmental and diagnostic purposes for high-stakes decisions and to improve instructors' teaching effectiveness (Annan et al., 2013; Fraile & Bosch-Morell, 2015; Ramsden, 2003; Setari, Lee, & Bradley, 2016; Zabaleta, 2007). Research on the outcome of SETs has frequently focused on the role
https://doi.org/10.1016/j.teln.2018.01.002 1557-3087/© 2017 Published by Elsevier Inc. on behalf of Organization for Associate Degree Nursing.
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
of course-related characteristics, such as class size and time, (Dziuban & Moskal, 2011; Tobin, 2017; Tuazon, 2015) and instructor's individual characteristics, such as gender and years of teaching (Carbone et al., 2015; Morgan, 2016). However, more attention is needed on the instructional characteristics (e.g., assessment and learning environment) that can be sensibly and instantly adopted by instructors than on class and instructor characteristics that are generally not under the control of instructors (Cho, Otani, & Kim, 2014; Otani, Kim, & Cho, 2012). Higher education institutions have been urged to reexamine the effective use of the routinely collected SET data at the end of each semester, make a conscious effort to improve teaching at the program, departmental, and college levels, and disseminate the outcomes and plan a timely response to student needs (Griffin & Cook, 2009; Wolfer & Johnson, 2003). Debates over the validity and reliability of SET have been documented, and the usefulness of SET in improving teaching quality and student learning has been questioned (Braga, 2014; Emery, Kramer, & Tian, 2003; Clayson & Haley, 2011; Morley, 2014; Wong & Moni, 2014; Zabaleta, 2007). However, other researchers are confident that SET is an intuitive tool to assess students' learning experiences and that course evaluations matter to improve instructors' teaching effectiveness (Carbone et al., 2015). Researchers have highlighted efforts to use other forms of inputs to validate the teaching effectiveness of university instructors. However, the results of previous studies concluded that SET is widely accepted as the most effective and efficient indicator of teaching effectiveness of university instructors and in decisions on merit pay, promotion and tenure, and teaching improvement (Agbetsiafa, 2010; Hendry, Lyon, & Henderson-Smart, 2007; Iqbal, 2013). Factors Affecting SET SET can be influenced by numerous course-related variables, such as class size, class schedule, curricular area, prior interest in the course topic, workload, difficulty, whether a course is required or elective, challenge level of the course, instructor-related characteristics, and online versus traditional course format (Dziuban et al., 2011; Johnson, Narayanan, & Sawaya, 2013; Tobin, 2017; Zabaleta, 2007). Instructors' teaching effectiveness can be affected by other factors, including motivation and grades (Griffin, Hilton, Plummer, & Barret, 2014; Matos-Diaz & Ragan, 2010; Pintrich, 2003; Stehle, Spinath, & Kadmon, 2012). However, these variables may not be under the control of faculty. Course-related variables are usually arranged through administrative decisions and curriculum changes. Moreover, researchers may be able to continue to find more external variables affecting the results of SET and student learning because teaching is a dynamic enterprise. Given the increased emphasis on teaching quality and accountability in the era of uncertainty surrounding SET, instructors need to be proactive and open-minded to focus on aspects of teaching that they can control and improve to meet students' needs. Identifying the aspects of teaching that affect the overall rating of teaching is a beginning and crucial step toward more effectively improving teaching, thereby achieving more efficient utilization of routinely collected numeric SET data. Value and Task of SET While there is widespread skepticism about the summative purpose of SET, it is a tool to obtain students' feedback on their instructor's teaching practices (Griffin & Cook, 2009) that has the potential to gain formative feedback for instructors (Zabaleta, 2007) and demonstrate instructors' dedication to teaching and improvement (Surgenor, 2013). To date, researchers have explored the effectiveness of various measures and strategies of teaching improvement, such as external and internal reviews, course portfolio, peer
87
observations, and coaching, as well as SET (Iqbal, 2013; Wong & Moni, 2014). However, instructors believe that SET matters most in various high-stakes decisions and are resistant to other ways of demonstrating their teaching effectiveness, such as summative peer review of teaching because of efficiency and accessibility reasons (Iqbal, 2013). Nevertheless, in their comprehensive literature review on SET, Campbell, Steiner, and Gerdes (2005) found a lack of focus on aspects of teaching that can be controlled and readily improved by instructors. It is imperative to redirect the focus to improving various aspects of teaching based on the needs of students by systematically analyzing SET results. Faculty have expressed challenges in knowing how to respond to student evaluations (Spiller & Harris, 2013). It is possible to proactively identify urgent practical teaching actions through a systematic and purposeful analysis of a pattern among student responses in SET. Appropriate improvements in teaching practices can be made through identifying controllable variables, such as teaching strategies, assessment strategies, course objectives, classroom environment, and organization (Otani et al., 2012). However, to fully understand which controllable aspects of teaching affect the overall rating of the instructor's teaching performance as excellent, more research is needed on specific practical variables, including the syllabus, grading, preparation, individual assistance, motivation, practical application, relevance of the course content to future profession, critical thinking skills, and challenge. Interpretation of SET A statistical analysis using the numeric values of SET data can clearly distinguish between minimally significant and highly influential aspects of teaching perceived by students when they formulate their overall rating of the instructor's teaching as excellent. This identification process benefits from systematic data analysis and is not achievable by simply selecting improvement ideas by arbitrary decisions based on raw mean scores (Boysen, 2015). In one of the few such studies on identifying influential aspects of SET, Otani et al. (2012) reported that the clarity of instructors' explanation of the subject was the most influential aspect for students' rating of the instructor's overall teaching effectiveness. Other influential aspects included positive learning environment, the effective use of class time, and stimulating course materials. The current study is designed to reveal distinctive aspects of teaching in nursing, which are influential for nursing instructors to demonstrate their dedication to excellence in teaching. Excellence in Teaching in Nursing Excellence in teaching has been a topic of interest in nursing. According to the American Association of Colleagues of Nursing (2017), nursing programs have experienced a shortage of qualified faculty in a time of increased demand for professional registered nurses. Lack of emphasis on teaching excellence in nursing has been a continuous concern to meet the expectations and needs of nursing students (Johnson-Farmer & Frenn, 2009). To grow, nurture, and retain quality nursing faculty, researchers have emphasized the importance of student input in improving teaching and faculty awareness and responsiveness to students' evaluation of their teaching performance (Girija, 2012; Lerret & Frenn, 2011). Wood and Su (2017) outlined that excellent teachers are capable of creating motivational learning relationships, have expertise and skills in their disciplines and pedagogic approaches to encourage students' critical thinking, inspire students to have a desire for learning, and provide risk-free and positive learning environments. SET is capable of gathering student views on areas of dissatisfaction and excellence for these teaching characteristics of faculty and provides a foundation that guide and motivate university
88
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
instructors to strive for teaching excellence in order to provide quality nursing education that prospective nurses deserve. Hypotheses There is a gap in the literature regarding how students perceive various aspects of teaching in SET when they rate instructors as excellent overall. In a systematic literature review of the SET research in nursing programs, Annan et al. (2013) acknowledged a lack of SET studies in nursing and called for more systematic interpretation of SET results to make effective improvements in teaching. Based on the research on SET at higher education institutions, we developed hypotheses that were tested in the current study: (a) Nursing students combine each of the aspects of teaching to arrive at their overall evaluations of the instructor's teaching as excellent; and (b) Nursing students give more weight to certain aspects of teaching than others in assessing their overall rating of the excellence of the instructor's teaching. Methodology Participants and Setting Students in a department of nursing in a midwestern state university participated in the study. The SET data from eight academic semesters contain 2,211 student responses from Summer of 2014 to Fall of 2016. In all applicable nursing courses, the anonymous SET surveys were collected online at the end of each semester. To protect confidentiality, faculty and students' demographic backgrounds were not collected. The accessible student demographics of all students enrolled in the department of Nursing from Summer 2014 to Fall 2016 are as follows: An average of 652 students were enrolled in the nursing program each semester: 87 male (13.3%) and 565 female (86.7%). The average age of students was 26.2 years. The average number of students in each class is about 22. Full-time students make up 51.7% of the total students, and 48.3% are part-time. Regarding race and ethnicity, on average, 83.6% of the total population is White, followed by 5.8% Hispanic, 2.7% Asian, 1.6% African American, 1.6% of mixed race, 0.3% American Indian, and 0.1% others.
by Divoky and Rothermel (1988): dimension of delivery (Q3 and Q9), depth of knowledge (Q7 and Q10), interpersonal skills (Q5 and Q6), instructor's organization skill (Q1 and Q2), and relevance (Q4 and Q8; see Table 1). Consistent with the recommendation by Cashin and Downey (1992), the SET contains a global rating item (dependent variable) that indicates the overall evaluation of the instructor's teaching: Question 11: “Overall, I would rate this instructor as:” ranked on a 5-point Likert scale (5 = excellent, 4 = good, 3 = fair, 2 = poor, and 1 = very poor). The SET survey also used a 5-point Likert scale for the independent variables (5 = strongly agree, 4 = agree, 3 = neither agree nor disagree, 2 = disagree, and 1 = strongly disagree). Data Analysis For this study, we used logistic regression analysis to investigate the effect of the 10 multidimensional evaluation items on the ranking of an instructor's overall teaching as excellent, specifically to identify which aspects of teaching are more influential on the overall rating as excellent. The current study focused on the factors that led to the rating of excellent. In the nursing program, the ratings are mainly excellent versus good and fair. Poor and very poor can be considered as outliers. Based on this distribution, we recoded the dependent variable into two levels, excellent and not excellent (good, fair, poor, and very poor). A binary dependent variable was created by coding the excellent rating as 1 and not excellent as 0. A logistic regression model was used to examine the relative importance of the 10 aspects of teaching (independent variables) on the dependent variable (overall instructor rating item that is rated as excellent). The general model for the ith case is as follows: ln
n X P ðY ¼ 1Þ bi xi þ e; ¼aþ 1−P ðY ¼ 1Þ i¼1
where Y is whether overall SET is excellent (Y = 1) or not (Y = 0), a is the intercept, bi is a regression coefficient, xi is an experience of the ith aspect, and e is an error term. By definition, P(Y = 1)/(1−P (Y = 1) is the odds ratio (OR) of getting an excellent overall evaluation, OR N 1 means that the dependent variable Y is more likely to have the value 1, or in other words, the teaching evaluation is more likely to get a rating of excellent.
Instrument Results The nursing department in this study employed a departmentwide form with 11 evaluation items, including 10 multidimensional items (see Table 1). The SET in the nursing department was developed based on the recommendation of the center for teaching and learning on campus and meets the recommendation for a valid SET instrument as it measures five dimensions of teaching recommended
Descriptive Statistics Average students' responses for the independent variables (evaluation items) are generally high, ranging from 4.05 (SD = 1.117) to 4.42 (SD = 0.849; 5 = strongly agree and 1 = strongly disagree).
Table 1 Descriptive Statistics of SET Survey Items Survey items Independent variables Q1: The syllabus contains adequate information. Q2: Grading is clear and tied to key learning objectives. Q3: Concepts are presented in a manner that helps me learn. Q4: Practical applications of course materials are discussed. Q5: My instructor provides individual assistance when asked. Q6: My instructor creates an environment for mutual respect. Q7: My critical thinking skills improved because of this course. Q8: My instructor attempts to relate my present learning to work in my future profession. Q9: My instructor makes learning interesting and motivates students to learn. Q10: In this course, I always felt challenged and motivated to learn. Dependent variable Q11: Overall, I would rate this instructor as excellent–good–fair–poor–very poor
M
SD
N
4.36 4.30 4.09 4.30 4.31 4.36 4.14 4.42 4.15 4.05
.841 .944 1.100 .905 .952 .965 1.047 .849 1.105 1.117
2206 2209 2208 2209 2210 2206 2206 2202 2209 2208
4.14
.934
2207
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
The average overall evaluation (dependent variable) is 4.14 (5 = excellent and 1 = very poor), and its SD is 0.934. Table 1 provides basic descriptive statistics of variables collected in SET. Logistic Regression Analysis Table 2 presents the parameter estimate, standard error (SE), p value, OR, 95% confidence interval (CI) for the OR, and ranking order. Out of all IV's, five variables are statistically significant (Q9. My instructor makes learning interesting and motivates students to learn; Q5. My instructor provides individual assistance when asked; Q3. Concepts are presented in a manner that helps me learn; Q6. My instructor creates an environment of mutual respect; and Q4. Practical applications of course material are discussed), and they are all related to the OR of getting an excellent rating. For example, the OR for Q9 is 2.808, meaning if the rating for this item increases one unit, the OR of getting an excellent overall evaluation will be about 2.81 times higher before the increase of the independent variable when all other variables are held constant. The 95% CI of the independent variable (Q9; 2.112–3.733) means that we are 95% confident that this kind of increase in OR is at least 2.112 times. The information for the other independent variables can be interpreted in the same way. Discussion The purpose of the present study is to investigate the influence of multidimensional evaluation items on students' rating of the overall assessment of the teaching quality of their instructors as excellent in a nursing program. In particular, identifying influential items on the overall rating of instructor's teaching as excellent is a primary research focus. We systematically identified influential aspects on students' overall experience as excellent using logistic regression analysis in order to prioritize teaching practices for more effective and efficient improvement of instructors' teaching quality in a nursing department. As hypothesized, consistent with findings in previous studies on SET (Cashin et al., 1992; Cho et al., 2014; Otani et al., 2012), certain SET items are more influential on the students' overall rating of teaching as excellent. The most influential item on the rating of the overall evaluation of the nursing faculty's teaching as excellent is intrinsic motivation (Q9: My instructor makes learning interesting and motivates students to learn) followed by individual assistance, clear presentation of concepts, learning environment, and practical Table 2 Results of Logistic Regression Analysis with Dichotomous Overall Satisfaction as Dependent Variable (Overall, I Would Rate This Instructor as Excellent–Good–Fair–Poor– Very Poor) and Ranking Order of Influential Aspects of Teaching Variable
Estimate
SE
p
OR
95% CI for OR Lower Upper
Q1. Syllabus −0.109 Q2. Grading 0.159 Q3. Clear presentation 0.586 of concepts Q4. Practical 0.521 applications Q5. Individual 0.604 assistance Q6. Environment 0.572 Q7. Critical thinking 0.026 Q8. Relevance to −0.137 future profession Q9. Interesting and 1.032 motivating instructor Q10. Challenge 0.156 Constant −14.642
Rank order
0.146 .455 0.897 0.673 0.156 .306 1.173 0.865 0.134 0 1.797 1.381
1.194 1.591 2.337
3
0.167 .002 1.684 1.214
2.336
5
0.157 0
1.345
2.491
2
0.18 .002 1.772 1.245 0.161 .874 1.026 0.748 0.185 .46 0.872 0.607
2.524 1.407 1.254
4
0.145 0
2.808 2.112
3.733
1
0.145 .284 1.168 0.879 0.659 0 0
1.553
1.83
89
application (see Fig. 1). The results from the logistic regression analysis support the perspective of motivational science in learning and teaching contexts that upholds the importance of intrinsic motivation in student learning and understanding various factors (e.g., higher expectations for student performance, stimulating and interesting tasks, appropriate feedback, and opportunity to be successful but challenged) that affect students' motivation in classrooms (Pintrich, 2003). Moreover, the findings of the current study support Wood et al.'s (2017) description of excellent teachers as those who are able to motivate students, establish a safe learning environment, influence students' desire to learn with appropriate disciplinary knowledge and approaches, and have a supportive relationship with students. Our findings provide further evidence to support previous findings that a caring and positive classroom environment helps students to be motivated for learning and find learning interesting (Gillet, Berjot, Vallerand, & Amoura, 2012; León, Núñez, & Liew, 2015). This aspect has an influence on students' evaluation of their instructor's teaching effectiveness and student learning outcomes (Lizzio, Wilson, & Simons, 2002; Lovric et al., 2014). All these aspects of teaching can be controlled and facilitated by instructors. Nursing faculty can adopt more innovative teaching methods to arouse students' interest and motivation in learning the content, improve the course contents to become more practical and relevant to nursing students' needs, make a more systematic effort to create cooperative relationships, and offer more assistance as they strive to improve their teaching and the learning experiences of students (Papathanasiou, Tsaras, & Sarafis, 2014; Wolf, Bender, Beitz, Wieland, & Vito, 2004). The present study aimed to demonstrate how nursing programs may create a better learning experience for their students by identifying influential aspects of teaching so that instructors may better address their students' needs and strategically and systematically improve their teaching effectiveness to become excellent. Logistic regression analysis of SET data helps to identify which aspects of teaching are more influential on the overall experience of teaching as excellent. To date, very few studies have performed this analysis. Langbein and Snider (1999) tested the impact of SET ratings on the probability of retention and found that that poorly-rated courses negatively affect the retention of students in college. Teaching improvement using the systematic analysis of SET results may be a sensible and attainable effort faculty can make to achieve excellence in teaching and retain their students to accomplish successful graduation, rather than being passive and powerless because of variables (e.g., class, student, and instructor characteristics) that instructors usually have no control over. Our analysis identified five aspects that were not statistically influential on the overall excellence rating of the instructor's teaching: adequate syllabus, critical thinking skills, relevance to future profession, grading, and challenge. It is possible that the value of the syllabus is not significant to students when they evaluate the overall teaching quality of their instructors as excellent. Faculty view syllabi as a teaching and communication tool and take time to create a comprehensive and detailed syllabus, but students may not pay attention to items that faculty consider to be important information (Becker & Calhoon, 1999). Furthermore, there were unexpected findings that critical thinking skills and relevance of class content to future profession were not influential on students' rating of the instructors' overall teaching performance as excellent. These are highly emphasized and promoted in 21st century education, and health care professionals are especially required to have sophisticated critical thinking skills in order to address the dynamic and critical issues patients experience along with in-demand skills, such as problem-solving, collaboration, and interpersonal skills. Critical thinking skills are required to effectively care for different patient populations at different health priority levels. In fact, the importance of critical thinking in nursing practice
90
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
3
Odds Ratio (95%CI)
2.5 2 1.5 1 0.5 0 Q9. Motivation
Q5. Individual Assistance
Q3. Clear Presentation
Q6. Environment
Q4. Practical Application
Influential Aspects of Teaching Fig. 1. Relative importance of influential aspects on the overall rating as excellent.
has been reported, and various teaching strategies to promote critical thinking skills for nursing students have been introduced (Latif, Mohamed, Dahlan, & Mat Nor, 2016). In addition, critical thinking is a significant factor in nursing educational programs (Raymond, Profetto-McGrath, Myrick, & Strean, 2017). Nursing students might value other aspects of teaching as part of a positive learning experience, which might not necessarily fully reflect the main objectives of nursing programs. Another interesting finding is that grading practice did not affect the overall evaluation of teaching as excellent. There are mixed results about the relationship between grading and SET (Heckert, Latier, Ringwald-Burton, & Drazen, 2006). Grading has been found to influence students' evaluation of teaching effectiveness in disciplines, such as social work (Jirovec, Ramanathan, & Rosegrant-Alvarez, 1998) and public policy (Cho et al., 2014), but Griffin et al. (2014) reported a lack of correlation between grades and SET in religion classes and advised administrators to take caution not to misinterpret high grades and high SET as evidence of a cause and effect relationship between grades and SET. These inconsistent and unexpected results may be because of the nature of grading as a potential source of bias in higher education and may be related to unique perceptions students have about teaching and learning in different disciplines and students' main goal of enrolling in a higher education program, which necessitates the focus on learning practical applications rather than grades. Consistent with Griffin et al. (2014), the current finding that a clear grading practice tied to course objectives does not affect students' rating of their overall learning experience as excellent may be the result of students being unaware of the value of instructors' actions related to clarifying and linking grading to course objectives, which is a key component of a typical program evaluation report to demonstrate the quality of the program. In short, it is likely that students do not care about the same things faculty, administrators, and program accreditation evaluators care about. The current findings confirmed that numerical data in SET can enable instructors to clearly
see the aspects of teaching they need to focus on to meet the needs of their students in a timely manner and can be used as an effective tool by faculty in developing teaching excellence in nursing. It should be noted that student course evaluation is not the same as customer satisfaction. University instructors must be strategic and selective in how they reflect and implement the findings of the current research in their teaching practices because they have a comprehensive understanding of the mission and objectives of the program and courses and are committed to improve their pedagogic expertise in nursing education. Limitations and Implications The current paper investigated the aspects of teaching that influence the overall rating of an instructor's teaching as excellent. Although the results are promising, this study has several limitations, including the single-program nature of the sample and the sampling method. We used a sample in one nursing department, which would limit the generalizability of the study results. To generate more generalizable results, this SET study should be replicated in other nursing programs to investigate the levels of influence and patterns of various aspects of teaching that are valued by nursing students. Such investigation will certainly be challenging (Wolfer & Johnson, 2003) but is critical and timely to give information about more effective and preferred learning experiences of nursing students. Another critical requirement for future research is to obtain the perspectives of the instructors to discover the mismatches between teachers and students in aspects of teaching they value as excellence in teaching and delivering nursing knowledge and skills. Because of the anonymous nature of SET, external factors, such as student and course characteristics and instructor information were not obtained, but we are aware of the possibility that these factors might have influenced the overall rating of the instructors' teaching as excellent in the current study. However, the focus of the current study was on the controllable instructional variables that can be
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
constructively approached and managed by instructors, and the present results are consistent with previous findings on the power of motivation in student learning experiences. Nursing student evaluations are complex because of the overlap of learning and application in a variety of learning settings. Future research in nursing SET should make use of accessible characteristics of students, classes, and instructors and explore their relationships with multidimensional aspects of teaching and the overall rating of instructors as excellent in nursing programs (Annan et al., 2013). The dependent variable (i.e., the overall rating of instructors' teaching as excellent), was dichotomous (excellent or not), as the current study focused on examining the relative importance of the 10 aspects on the dependent variable. A logistic regression analysis was appropriate for the purpose of the study. The results of the analysis made it evident that simple comparisons of raw mean scores would not allow instructors to identify influential factors on the overall evaluation of their teaching as excellent. For example, the course syllabus is not an influential aspect of teaching on the overall rating as excellent, but this aspect received one of the highest mean scores. Nevertheless, focusing only on the two categories of strongly agree (5) or not (4, 3, 2, and 1) in the dependent variables may result in loss of information even though there is only a small number of ratings of 3, 2, or 1. In order to make full use of all the information, an ordinal logistic regression model may be considered. Conclusion The contribution of the present study can be found in its focus on influential yet controllable aspects of teaching that have a significant impact on the overall rating of instructors' teaching performance as excellent in a university nursing program. This study reported nursing students' unique focus on certain aspects of teaching when they evaluate whether their instructor's overall performance was excellent or not. Nursing students indicated that they placed different levels of importance on certain aspects of teaching, including motivation, individual assistance, clear presentation of concepts, learning environment, and practical application. When students assessed the overall excellence of their instructors' teaching, they were not concerned about the quality of a syllabus, relevance of the course content to their future profession, critical thinking skills, grading, and challenge in a course. Apparently, aspects of teaching that typically receive a great deal of faculty time and effort and are treasured by instructors as a way to communicate course expectations and objectives are not perceived at the same level by students. By being consumers of the results of SET who systematically analyze the results and prioritize the aspects of teaching to demonstrate excellence in teaching, faculty can provide students with better learning opportunities in a positive and motivating learning environment, adopt diverse teaching practices to emphasize various aspects of teaching that are critical for student learning and progress, and help students to be confident in the skills they learned so that they can effectively and successfully work in interprofessional health care settings and provide the highest quality of care to their future patients. Acknowledgments The authors would like to thank Regina Gordon and Adam Dircksen for their assistance with preparation of the aggregated data. References Agbetsiafa, D. (2010). Evaluating effective teaching in college level economics using student ratings of instruction: A factor analytic approach. Journal of College Teaching and Learning, 7(5), 57–66.
91
Al-Maamari, F. (2015). Response rate and teaching effectiveness in institutional student evaluation of teaching: A multiple linear regression study. Higher Education Studies, 5(6), 9–20. American Association of Colleges of Nursing (2017). Nursing faculty shortage fact sheet. Washington, DC: Rosseter, R [Retrieved from http://www.aacnnursing.org/ Portals/42/News/Factsheets/Faculty-Shortage-Factsheet-2017.pdf]. Annan, S. L., Tratnack, S., Rubenstein, C., Metzler-Sawin, E., & Hulton, L. (2013). An integrative review of student evaluations of teaching: Implications for evaluation of nursing faculty. Journal of Professional Nursing, 29(5), e10–e24. https://doi.org/10. 1016/j.profnurs.2013.06.004. Becker, A. H., & Calhoon, S. K. (1999). What introductory psychology students attend to on a course syllabus. Teaching of Psychology, 26(1), 6–11. Boysen, G. A. (2015). Uses and misuses of student evaluations of teaching: The interpretation of differences in teaching evaluation means irrespective of statistical information. Teaching of Psychology, 42(2), 109–118. https://doi.org/10.1177/ 0098628315569922. Braga, M., Paccagnella, M., & Pellizzari, M. (2014). Evaluating students' evaluations of professors. Economics of Education Review, 4171–4188. https://doi.org/10.1016/j. econedurev.2014.04.002. Campbell, H. E., Steiner, S., & Gerdes, K. (2005). Student evaluations of teaching. Journal of Public Affairs Education, 11(3), 211–231. Carbone, A., Ross, B., Phelan, L., Lindsay, K., Drew, S., Stoney, S., & Cottman, C. (2015). Course evaluation matters: Improving students' learning experiences with a peer-assisted teaching programme. Assessment & Evaluation in Higher Education, 40(2), 165–180. https://doi.org/10.1080/02602938.2014.895894. Cashin, W. E., & Downey, R. G. (1992). Using global student rating items for summative evaluation. Journal of Educational Psychology, 84(4), 563–572. Cho, J., Otani, K., & Kim, B. J. (2014). Differences in student evaluations of limited-term lecturers and full-time faculty. Journal on Excellence in College Teaching, 25(2), 5–24. Clayson, D. E., & Haley, D. A. (2011). Are students telling us the truth? A critical look at the student evaluation of teaching. Marketing Education Review, 21(2), 101–112. https://doi.org/10.2753/MER1052-8008210201. Divoky, J. J., & Rothermel, M. A. (1988). Student perceptions of the relative importance of dimensions of teaching performance across a type of class. Educational Research Quarterly, 12(3), 40–45. Dziuban, C., & Moskal, P. (2011). A course is a course is a course: Factor invariance in student evaluation of online, blended and face-to-face learning environments. The Internet and Higher Education, 14(4), 236–241. https://doi.org/10.1016/j.iheduc. 2011.05.003. Emery, C. R., Kramer, T. R., & Tian, R. G. (2003). Return to academic standards: A critique of student evaluations of teaching effectiveness. Quality Assurance in Education, 11(1), 37–46. Fraile, R., & Bosch-Morell, F. (2015). Considering teaching history and calculating confidence intervals in student evaluations of teaching quality. Higher Education, 70 (1), 55–72. https://doi.org/10.1007/s10734-014-9823-0. Gillet, N., Berjot, S., Vallerand, R. J., & Amoura, S. (2012). The role of autonomy support and motivation in the prediction of interest and dropout intentions in sport and education settings. Basic and Applied Social Psychology, 34(3), 278–286. https:// doi.org/10.1080/01973533.2012.674754. Griffin, A., & Cook, V. (2009). Acting on evaluation: Twelve tips from a national conference on student evaluations. Medical Teacher, 31, 101–104. Griffin, T. J., Hilton, J., III, Plummer, K., & Barret, D. (2014). Correlation between grade point averages and student evaluation of teaching scores: Taking a closer look. Assessment & Evaluation in Higher Education, 39(3), 339–348. https://doi.org/10. 1080/02602938.2013.831809. Girija, K. (2012). Effective clinical instructor– a step toward excellence in clinical teaching. International Journal of Nursing Education, 4(1), 25–27. Heckert, T. M., Latier, A., Ringwald-Burton, A., & Drazen, C. (2006). Relations among student effort, perceived class difficulty appropriateness, and student evaluations of teaching: Is it possible to “buy” better evaluations through lenient grading? College Student Journal, 40(3), 588–596. Hendry, G. D., Lyon, P. M., & Henderson-Smart, C. (2007). Teachers' approaches to teaching and responses to student evaluation in a problem-based medical program. Assessment & Evaluation in Higher Education, 32(2), 143–157. https://doi. org/10.1080/02602930600801894. Jirovec, R. L., Ramanathan, C. S., & Alvarez, A. R. (1998). Course evaluation: What are social work students telling us about teaching effectiveness? Journal of Social Work Education, 34(2), 229–236. Iqbal, I. (2013). Academics' resistance to summative peer review of teaching: Questionable rewards and the importance of student evaluations. Teaching in Higher Education, 18(5), 557–569. https://doi.org/10.1080/13562517.2013.764863. Johnson-Farmer, B., & Frenn, M. (2009). Teaching excellence: What great teachers teach us. Journal of Professional Nursing, 25(5), 267–272. https://doi.org/10.1016/ j.profnurs.2009.01.020. Johnson, M. D., Narayanan, A., & Sawaya, W. J. (2013). Effects of course and instructor characteristics on student evaluation of teaching across a college of engineering. Journal of Engineering Education, 2(102), 289–318. https://doi.org/10.1002/jee. 20013. Latif, R. A., Mohamed, R., Dahlan, A., & Mat Nor, M. Z. (2016). Concept mapping as a teaching tool on critical thinking skills and academic performance of diploma nursing students. Education in Medicine Journal, 8(1), 67–74. https://doi.org/10. 5959/eimj.v8i1.406. Langbein, L. I., & Snider, K. (1999). The impact of teaching on retention: Some quantitative evidence. Social Science Quarterly, 80(3), 457–472.
92
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
León, J., Núñez, J. L., & Liew, J. (2015). Self-determination and STEM education: Effects of autonomy, motivation, and self-regulated learning on high school math achievement. Learning & Individual Differences, 43, 156–163. https://doi.org/10. 1016/j.lindif. 2015.08.017. Lerret, S. M., & Frenn, M. (2011). Challenge with care: Reflections on teaching excellence. Journal of Professional Nursing, 27(6), 378–384. https://doi.org/10.1016/j. profnurs. 2011.04.014. Lizzio, A., Wilson, K., & Simons, R. (2002). University students' perceptions of the learning environment and academic outcomes: Implications for theory and practice. Studies in Higher Education, 27(1), 27–52. Lovric, R., Nrlic, N., Barac, I., Pluzaric, J., Puseljic, S., Berecki, I., & Radic, R. (2014). Specificities and differences in nursing students' perceptions of nursing clinical faculties' competencies. Journal of Professional Nursing, 30, 406–417. https://doi.org/ 10.1016/j.profnurs.2014.03.005. Matos-Diaz, H., & Ragan, J. F. (2010). Do student evaluations of teaching depend on the distribution of expected grade? Education Economics, 18(3), 317–330. https://doi. org/10.1080/09645290903109444. Morgan, H. K., Purkiss, J. A., Porter, A. C., Lypson, M. L., Santen, S. A., Christner, J. G., & Hammoud, M. M. (2016). Student evaluation of faculty physicians: Gender differences in teaching evaluations. Journal of Women's Health, 25(5), 453–456. https:// doi.org/10.1089/jwh.2015.5475. Morley, D. (2014). Assessing the reliability of student evaluations of teaching: Choosing the right coefficient. Assessment & Evaluation in Higher Education, 39(2), 127–139. https://doi.org/10.1080/02602938.2013.796508. Oermann, M. (2017). Student evaluations of teaching: There is more to course evaluations than student ratings. Nurse Educator, 42, 55–56. https://doi.org/10.1097/ NNE0000000000000366. Otani, K., Kim, B. J., & Cho, J. (2012). Student course evaluation (SET) in higher education: How to use student course evaluation more effectively and efficiently in public affairs education. Journal of Public Affairs Education, 18(3), 531–544. Papathanasiou, I., Tsaras, K., & Sarafis, P. (2014). Views and perceptions of nursing students on their clinical learning environment: Teaching and learning. Nurse Education Today, 34(1), 57–60. https://doi.org/10.1016/j.nedt.2013.02.007. Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95(4), 667–686. Ramsden, P. (2003). Learning to teach in higher education. London, UK: Taylor & Francis. Raymond, C., Profetto-McGrath, J., Myrick, F., & Strean, W. B. (2017). An integrative review of the concealed connection: Nurse educators' critical thinking. The Journal of
Nursing Education, 56(11), 648–654. https://doi.org/10.3928/0148483420171020-03. Setari, A. P., Lee, J., & Bradley, K. D. (2016). A psychometric approach to the validation of a student evaluation of teaching instrument. Studies in Educational Evaluation, 51, 77–87. https://doi.org/10.1016/j.stueduc.2016.09.006. Spiller, D., & Harris, T. (2013). Learning from evaluations: Probing the reality. Issues in Educational Research, 23(2), 258–268. Stehle, S., Spinath, B., & Kadmon, M. (2012). Measuring teaching effectiveness: Correspondence between students' evaluations of teaching and different measures of student learning. Research in Higher Education, 53(8), 888–904. https://doi.org/ 10.1007/s11162-012-9260-9. Surgenor, P. (2013). Obstacles and opportunities: Addressing the growing pains of summative student evaluation of teaching. Assessment & Evaluation in Higher Education, 38(3), 363–376. https://doi.org/10.1080/02602938.2011.635247. Tobin, R. R. (2017). Too early for physics? Effect of class meeting time on student evaluations of teaching in introductory physics. Physics Teacher, 55(5), 276–279. Tuazon, A. A. (2015). Comparison of core curriculum and distribution models of general education: Student evaluation of general education social science courses at the University of the Philippines Diliman. Philippine Social Sciences Review, 67(1), 17–32. Valiga, T. (2017). Do schools of nursing truly value excellence in teaching? Actions speaks louder than words. Journal of Nursing Education, 56(9), 519–520. https:// doi.org/10.3928/01484834-20170817-01. Wolf, Z. R., Bender, P. J., Beitz, J. M., Wieland, D. M., & Vito, K. O. (2004). Strengths and weaknesses of faculty teaching performance reported by undergraduate and graduate nursing students: A descriptive study. Journal of Professional Nursing, 20(2), 118–128. https://doi.org/10.1016/j.profnurs.2004.03.003. Wolfer, T., & Johnson, M. M. (2003). Re-evaluating student evaluation of teaching: The teaching evaluation form. Journal of Social Work Education, 39(1), 111–121. Wong, W. Y., & Moni, K. (2014). Teachers' perceptions of and responses to student evaluation of teaching: Purposes and uses in clinical education. Assessment & Evaluation in Higher Education, 39(4), 397–411. https://doi.org/10.1080/ 02602938.2013.844222. Wood, M., & Su, F. (2017). What makes an excellent lecturer? Academics' perspectives on the discourse of ‘teaching excellence’ in higher education. Teaching in Higher Education, 22(4), 451–466. https://doi.org/10.1080/13562517.2017.1301911. Zabaleta, F. (2007). The use and misuse of student evaluations of teaching. Teaching in Higher Education, 12(1), 55–76.
Our reference: TELN 531
P-authorquery-v11
AUTHOR QUERY FORM Journal: TELN
Please e-mail your responses and any corrections to:
Article Number: 531
E-mail: [email protected]
Dear Author, Please check your proof carefully and mark all corrections at the appropriate place in the proof (e.g., by using on-screen annotation in the PDF file) or compile them in a separate list. Note: if you opt to annotate the file with software other than Adobe Reader then please also highlight the appropriate place in the PDF file. To ensure fast publication of your paper please return your corrections within 48 hours. For correction or revision of any artwork, please consult http://www.elsevier.com/artworkinstructions. We were unable to process your file(s) fully electronically and have proceeded by Scanning (parts of) your article
Rekeying (parts of) your article
Scanning the artwork
Any queries or remarks that have arisen during the processing of your manuscript are listed below and highlighted by flags in the proof. Click on the ‘Q’ link to go to the location in the proof.
Location in article
Query / Remark: click on the Q link to go Please insert your reply or correction at the corresponding line in the proof
Q1
The author names have been tagged as given names and surnames (surnames are highlighted in teal color). Please confirm if they have been identified correctly.
Q2
Please provide city and postal codes for all affiliations.
Q3
Ref. "Frail & Bosch-Morell, 2015" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q4
The citation "Carbone, 2015" has been changed to "Carbone et al., 2015" to match the author name/ date in the reference list. Please check if the change is fine in this occurrence and modify the subsequent occurrences, if necessary.
Q5
Ref. "Morgan, 2015" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q6
Ref. "Braga, 2014" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q7
Ref. "Dziuban et al., 2011" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q8
Ref. "Johnson, Narayanan, & Sawaya, 2013" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q9
Ref. "Iqbal, 2012" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q10
Ref. "Johnson-Farmer & Frenn, 2009" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Page 1 of 2
Our reference: TELN 531
P-authorquery-v11
Q11
Ref. "Girija, 2012" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q12
"CI" was inserted here to mean as "confidence interval." Please check if correct.
Q13
Refs. "Cashin et al., 1992", "Wood et al.’s (2017)" are cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q14
Ref. "Lanbein and Snider (1999)" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q15
Ref. "Jirovec, Ramanathan, & Rosegrant-Alvarez, 1998" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q16
Uncited references: This section comprises references that occur in the reference list but not in the body of the text. Please position each reference in the text or, alternatively, delete it. Thank you.
Q17
Please supply last page. Please check this box if you have no corrections to make to the PDF file.
□
Thank you for your assistance.
Page 2 of 2
Contents lists available at ScienceDirect
Teaching and Learning in Nursing journal homepage: www.jtln.org
Influential Factors of Student Evaluations of Teaching in a Nursing Program Jeong IL Cho a,⁎, Kawther Hamash b, Koichiro Otani c, Nila Reimer b, Yihao Deng d a
Department of Professional Studies, Purdue University Fort Wayne, 468, USA Department of Nursing, Indiana University Fort Wayne, 468, USA Department of Public Policy, Purdue University Fort Wayne, 468, USA d Department of Mathematical Sciences, Purdue University Fort Wayne, 468, USA b c
a r t i c l e
i n f o
Article history: Accepted 6 January 2018 Available online xxxx Keywords: Student evaluation of teaching Teaching excellence Nursing students Motivation SET
a b s t r a c t We used a logistic regression model to analyze data from routinely collected student evaluation of teaching from 2,211 nursing students over 8 semesters. The results showed that not all aspects of teaching are equally influential in their effects on students' overall rating of instructors' teaching as excellent. Influential aspects include motivation, individual assistance, presentation of concepts, environment, and practical application. The most influential evaluation aspect is motivation. © 2017 Published by Elsevier Inc. on behalf of Organization for Associate Degree Nursing.
Introduction Nursing educators have voiced a significant issue with the student evaluation of teaching (SET) and called for more systematic studies on the effective use of SET in nursing education programs (Annan, Tratnack, Rubenstein, Metzler-Sawin, & Hulton, 2013). Valiga (2017) called for action in achieving and sustaining excellence in teaching in nursing through improving the pedagogical expertise of faculty and using science in teaching and learning to design specific learning experiences to benefit nursing students for the complex and challenging future of nursing. More research should be directed to identifying nursing students' needs and exploring the teaching effectiveness of nursing faculty (Lovric et al., 2014). Even though there have been debates on the usability, reliability, and validity of SET in nursing education (Oermann, 2017), it is perceived as a critical and logical tool faculty and administrators can use to understand students' constantly changing needs and evaluation of faculty's instruction and teaching effectiveness (Agbetsiafa, 2010). Findings about how students evaluate the teaching effectiveness of faculty as excellently drawn from SET have a potential to be informative and practical in departmental- and program-level discussions among faculty as
⁎ Corresponding author. Tel.: +1 260 481 6454; fax: +1 260 481 5408. E-mail address: [email protected] (J. Cho).
part of the professional development activities to achieve teaching excellence in evolving nursing education. We investigated the influence of multidimensional evaluation aspects of SET on the overall rating as excellent with the intention of providing insights about the quality of nursing instructors' instructional practices perceived by students and the significant impact of certain instructional activities on students' overall evaluation of teaching excellence of their instructors. The systematic approach of the current study in the analysis of the routinely collected SET data at a midwestern state university responds to the continuing concern of faculty about how useful SET can be for the evaluation of their teaching effectiveness (Al-Maamari, 2015) and highlights the qualities that students value in excellent nursing instructors. Literature Review Student Evaluations of Teaching in Higher Education Institutions Student evaluations of teaching (SET) in higher education have received increasing administrative, instructional, and research attention because of the significant impact they have on judgmental and diagnostic purposes for high-stakes decisions and to improve instructors' teaching effectiveness (Annan et al., 2013; Fraile & Bosch-Morell, 2015; Ramsden, 2003; Setari, Lee, & Bradley, 2016; Zabaleta, 2007). Research on the outcome of SETs has frequently focused on the role
https://doi.org/10.1016/j.teln.2018.01.002 1557-3087/© 2017 Published by Elsevier Inc. on behalf of Organization for Associate Degree Nursing.
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
of course-related characteristics, such as class size and time, (Dziuban & Moskal, 2011; Tobin, 2017; Tuazon, 2015) and instructor's individual characteristics, such as gender and years of teaching (Carbone et al., 2015; Morgan, 2016). However, more attention is needed on the instructional characteristics (e.g., assessment and learning environment) that can be sensibly and instantly adopted by instructors than on class and instructor characteristics that are generally not under the control of instructors (Cho, Otani, & Kim, 2014; Otani, Kim, & Cho, 2012). Higher education institutions have been urged to reexamine the effective use of the routinely collected SET data at the end of each semester, make a conscious effort to improve teaching at the program, departmental, and college levels, and disseminate the outcomes and plan a timely response to student needs (Griffin & Cook, 2009; Wolfer & Johnson, 2003). Debates over the validity and reliability of SET have been documented, and the usefulness of SET in improving teaching quality and student learning has been questioned (Braga, 2014; Emery, Kramer, & Tian, 2003; Clayson & Haley, 2011; Morley, 2014; Wong & Moni, 2014; Zabaleta, 2007). However, other researchers are confident that SET is an intuitive tool to assess students' learning experiences and that course evaluations matter to improve instructors' teaching effectiveness (Carbone et al., 2015). Researchers have highlighted efforts to use other forms of inputs to validate the teaching effectiveness of university instructors. However, the results of previous studies concluded that SET is widely accepted as the most effective and efficient indicator of teaching effectiveness of university instructors and in decisions on merit pay, promotion and tenure, and teaching improvement (Agbetsiafa, 2010; Hendry, Lyon, & Henderson-Smart, 2007; Iqbal, 2013). Factors Affecting SET SET can be influenced by numerous course-related variables, such as class size, class schedule, curricular area, prior interest in the course topic, workload, difficulty, whether a course is required or elective, challenge level of the course, instructor-related characteristics, and online versus traditional course format (Dziuban et al., 2011; Johnson, Narayanan, & Sawaya, 2013; Tobin, 2017; Zabaleta, 2007). Instructors' teaching effectiveness can be affected by other factors, including motivation and grades (Griffin, Hilton, Plummer, & Barret, 2014; Matos-Diaz & Ragan, 2010; Pintrich, 2003; Stehle, Spinath, & Kadmon, 2012). However, these variables may not be under the control of faculty. Course-related variables are usually arranged through administrative decisions and curriculum changes. Moreover, researchers may be able to continue to find more external variables affecting the results of SET and student learning because teaching is a dynamic enterprise. Given the increased emphasis on teaching quality and accountability in the era of uncertainty surrounding SET, instructors need to be proactive and open-minded to focus on aspects of teaching that they can control and improve to meet students' needs. Identifying the aspects of teaching that affect the overall rating of teaching is a beginning and crucial step toward more effectively improving teaching, thereby achieving more efficient utilization of routinely collected numeric SET data. Value and Task of SET While there is widespread skepticism about the summative purpose of SET, it is a tool to obtain students' feedback on their instructor's teaching practices (Griffin & Cook, 2009) that has the potential to gain formative feedback for instructors (Zabaleta, 2007) and demonstrate instructors' dedication to teaching and improvement (Surgenor, 2013). To date, researchers have explored the effectiveness of various measures and strategies of teaching improvement, such as external and internal reviews, course portfolio, peer
87
observations, and coaching, as well as SET (Iqbal, 2013; Wong & Moni, 2014). However, instructors believe that SET matters most in various high-stakes decisions and are resistant to other ways of demonstrating their teaching effectiveness, such as summative peer review of teaching because of efficiency and accessibility reasons (Iqbal, 2013). Nevertheless, in their comprehensive literature review on SET, Campbell, Steiner, and Gerdes (2005) found a lack of focus on aspects of teaching that can be controlled and readily improved by instructors. It is imperative to redirect the focus to improving various aspects of teaching based on the needs of students by systematically analyzing SET results. Faculty have expressed challenges in knowing how to respond to student evaluations (Spiller & Harris, 2013). It is possible to proactively identify urgent practical teaching actions through a systematic and purposeful analysis of a pattern among student responses in SET. Appropriate improvements in teaching practices can be made through identifying controllable variables, such as teaching strategies, assessment strategies, course objectives, classroom environment, and organization (Otani et al., 2012). However, to fully understand which controllable aspects of teaching affect the overall rating of the instructor's teaching performance as excellent, more research is needed on specific practical variables, including the syllabus, grading, preparation, individual assistance, motivation, practical application, relevance of the course content to future profession, critical thinking skills, and challenge. Interpretation of SET A statistical analysis using the numeric values of SET data can clearly distinguish between minimally significant and highly influential aspects of teaching perceived by students when they formulate their overall rating of the instructor's teaching as excellent. This identification process benefits from systematic data analysis and is not achievable by simply selecting improvement ideas by arbitrary decisions based on raw mean scores (Boysen, 2015). In one of the few such studies on identifying influential aspects of SET, Otani et al. (2012) reported that the clarity of instructors' explanation of the subject was the most influential aspect for students' rating of the instructor's overall teaching effectiveness. Other influential aspects included positive learning environment, the effective use of class time, and stimulating course materials. The current study is designed to reveal distinctive aspects of teaching in nursing, which are influential for nursing instructors to demonstrate their dedication to excellence in teaching. Excellence in Teaching in Nursing Excellence in teaching has been a topic of interest in nursing. According to the American Association of Colleagues of Nursing (2017), nursing programs have experienced a shortage of qualified faculty in a time of increased demand for professional registered nurses. Lack of emphasis on teaching excellence in nursing has been a continuous concern to meet the expectations and needs of nursing students (Johnson-Farmer & Frenn, 2009). To grow, nurture, and retain quality nursing faculty, researchers have emphasized the importance of student input in improving teaching and faculty awareness and responsiveness to students' evaluation of their teaching performance (Girija, 2012; Lerret & Frenn, 2011). Wood and Su (2017) outlined that excellent teachers are capable of creating motivational learning relationships, have expertise and skills in their disciplines and pedagogic approaches to encourage students' critical thinking, inspire students to have a desire for learning, and provide risk-free and positive learning environments. SET is capable of gathering student views on areas of dissatisfaction and excellence for these teaching characteristics of faculty and provides a foundation that guide and motivate university
88
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
instructors to strive for teaching excellence in order to provide quality nursing education that prospective nurses deserve. Hypotheses There is a gap in the literature regarding how students perceive various aspects of teaching in SET when they rate instructors as excellent overall. In a systematic literature review of the SET research in nursing programs, Annan et al. (2013) acknowledged a lack of SET studies in nursing and called for more systematic interpretation of SET results to make effective improvements in teaching. Based on the research on SET at higher education institutions, we developed hypotheses that were tested in the current study: (a) Nursing students combine each of the aspects of teaching to arrive at their overall evaluations of the instructor's teaching as excellent; and (b) Nursing students give more weight to certain aspects of teaching than others in assessing their overall rating of the excellence of the instructor's teaching. Methodology Participants and Setting Students in a department of nursing in a midwestern state university participated in the study. The SET data from eight academic semesters contain 2,211 student responses from Summer of 2014 to Fall of 2016. In all applicable nursing courses, the anonymous SET surveys were collected online at the end of each semester. To protect confidentiality, faculty and students' demographic backgrounds were not collected. The accessible student demographics of all students enrolled in the department of Nursing from Summer 2014 to Fall 2016 are as follows: An average of 652 students were enrolled in the nursing program each semester: 87 male (13.3%) and 565 female (86.7%). The average age of students was 26.2 years. The average number of students in each class is about 22. Full-time students make up 51.7% of the total students, and 48.3% are part-time. Regarding race and ethnicity, on average, 83.6% of the total population is White, followed by 5.8% Hispanic, 2.7% Asian, 1.6% African American, 1.6% of mixed race, 0.3% American Indian, and 0.1% others.
by Divoky and Rothermel (1988): dimension of delivery (Q3 and Q9), depth of knowledge (Q7 and Q10), interpersonal skills (Q5 and Q6), instructor's organization skill (Q1 and Q2), and relevance (Q4 and Q8; see Table 1). Consistent with the recommendation by Cashin and Downey (1992), the SET contains a global rating item (dependent variable) that indicates the overall evaluation of the instructor's teaching: Question 11: “Overall, I would rate this instructor as:” ranked on a 5-point Likert scale (5 = excellent, 4 = good, 3 = fair, 2 = poor, and 1 = very poor). The SET survey also used a 5-point Likert scale for the independent variables (5 = strongly agree, 4 = agree, 3 = neither agree nor disagree, 2 = disagree, and 1 = strongly disagree). Data Analysis For this study, we used logistic regression analysis to investigate the effect of the 10 multidimensional evaluation items on the ranking of an instructor's overall teaching as excellent, specifically to identify which aspects of teaching are more influential on the overall rating as excellent. The current study focused on the factors that led to the rating of excellent. In the nursing program, the ratings are mainly excellent versus good and fair. Poor and very poor can be considered as outliers. Based on this distribution, we recoded the dependent variable into two levels, excellent and not excellent (good, fair, poor, and very poor). A binary dependent variable was created by coding the excellent rating as 1 and not excellent as 0. A logistic regression model was used to examine the relative importance of the 10 aspects of teaching (independent variables) on the dependent variable (overall instructor rating item that is rated as excellent). The general model for the ith case is as follows: ln
n X P ðY ¼ 1Þ bi xi þ e; ¼aþ 1−P ðY ¼ 1Þ i¼1
where Y is whether overall SET is excellent (Y = 1) or not (Y = 0), a is the intercept, bi is a regression coefficient, xi is an experience of the ith aspect, and e is an error term. By definition, P(Y = 1)/(1−P (Y = 1) is the odds ratio (OR) of getting an excellent overall evaluation, OR N 1 means that the dependent variable Y is more likely to have the value 1, or in other words, the teaching evaluation is more likely to get a rating of excellent.
Instrument Results The nursing department in this study employed a departmentwide form with 11 evaluation items, including 10 multidimensional items (see Table 1). The SET in the nursing department was developed based on the recommendation of the center for teaching and learning on campus and meets the recommendation for a valid SET instrument as it measures five dimensions of teaching recommended
Descriptive Statistics Average students' responses for the independent variables (evaluation items) are generally high, ranging from 4.05 (SD = 1.117) to 4.42 (SD = 0.849; 5 = strongly agree and 1 = strongly disagree).
Table 1 Descriptive Statistics of SET Survey Items Survey items Independent variables Q1: The syllabus contains adequate information. Q2: Grading is clear and tied to key learning objectives. Q3: Concepts are presented in a manner that helps me learn. Q4: Practical applications of course materials are discussed. Q5: My instructor provides individual assistance when asked. Q6: My instructor creates an environment for mutual respect. Q7: My critical thinking skills improved because of this course. Q8: My instructor attempts to relate my present learning to work in my future profession. Q9: My instructor makes learning interesting and motivates students to learn. Q10: In this course, I always felt challenged and motivated to learn. Dependent variable Q11: Overall, I would rate this instructor as excellent–good–fair–poor–very poor
M
SD
N
4.36 4.30 4.09 4.30 4.31 4.36 4.14 4.42 4.15 4.05
.841 .944 1.100 .905 .952 .965 1.047 .849 1.105 1.117
2206 2209 2208 2209 2210 2206 2206 2202 2209 2208
4.14
.934
2207
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
The average overall evaluation (dependent variable) is 4.14 (5 = excellent and 1 = very poor), and its SD is 0.934. Table 1 provides basic descriptive statistics of variables collected in SET. Logistic Regression Analysis Table 2 presents the parameter estimate, standard error (SE), p value, OR, 95% confidence interval (CI) for the OR, and ranking order. Out of all IV's, five variables are statistically significant (Q9. My instructor makes learning interesting and motivates students to learn; Q5. My instructor provides individual assistance when asked; Q3. Concepts are presented in a manner that helps me learn; Q6. My instructor creates an environment of mutual respect; and Q4. Practical applications of course material are discussed), and they are all related to the OR of getting an excellent rating. For example, the OR for Q9 is 2.808, meaning if the rating for this item increases one unit, the OR of getting an excellent overall evaluation will be about 2.81 times higher before the increase of the independent variable when all other variables are held constant. The 95% CI of the independent variable (Q9; 2.112–3.733) means that we are 95% confident that this kind of increase in OR is at least 2.112 times. The information for the other independent variables can be interpreted in the same way. Discussion The purpose of the present study is to investigate the influence of multidimensional evaluation items on students' rating of the overall assessment of the teaching quality of their instructors as excellent in a nursing program. In particular, identifying influential items on the overall rating of instructor's teaching as excellent is a primary research focus. We systematically identified influential aspects on students' overall experience as excellent using logistic regression analysis in order to prioritize teaching practices for more effective and efficient improvement of instructors' teaching quality in a nursing department. As hypothesized, consistent with findings in previous studies on SET (Cashin et al., 1992; Cho et al., 2014; Otani et al., 2012), certain SET items are more influential on the students' overall rating of teaching as excellent. The most influential item on the rating of the overall evaluation of the nursing faculty's teaching as excellent is intrinsic motivation (Q9: My instructor makes learning interesting and motivates students to learn) followed by individual assistance, clear presentation of concepts, learning environment, and practical Table 2 Results of Logistic Regression Analysis with Dichotomous Overall Satisfaction as Dependent Variable (Overall, I Would Rate This Instructor as Excellent–Good–Fair–Poor– Very Poor) and Ranking Order of Influential Aspects of Teaching Variable
Estimate
SE
p
OR
95% CI for OR Lower Upper
Q1. Syllabus −0.109 Q2. Grading 0.159 Q3. Clear presentation 0.586 of concepts Q4. Practical 0.521 applications Q5. Individual 0.604 assistance Q6. Environment 0.572 Q7. Critical thinking 0.026 Q8. Relevance to −0.137 future profession Q9. Interesting and 1.032 motivating instructor Q10. Challenge 0.156 Constant −14.642
Rank order
0.146 .455 0.897 0.673 0.156 .306 1.173 0.865 0.134 0 1.797 1.381
1.194 1.591 2.337
3
0.167 .002 1.684 1.214
2.336
5
0.157 0
1.345
2.491
2
0.18 .002 1.772 1.245 0.161 .874 1.026 0.748 0.185 .46 0.872 0.607
2.524 1.407 1.254
4
0.145 0
2.808 2.112
3.733
1
0.145 .284 1.168 0.879 0.659 0 0
1.553
1.83
89
application (see Fig. 1). The results from the logistic regression analysis support the perspective of motivational science in learning and teaching contexts that upholds the importance of intrinsic motivation in student learning and understanding various factors (e.g., higher expectations for student performance, stimulating and interesting tasks, appropriate feedback, and opportunity to be successful but challenged) that affect students' motivation in classrooms (Pintrich, 2003). Moreover, the findings of the current study support Wood et al.'s (2017) description of excellent teachers as those who are able to motivate students, establish a safe learning environment, influence students' desire to learn with appropriate disciplinary knowledge and approaches, and have a supportive relationship with students. Our findings provide further evidence to support previous findings that a caring and positive classroom environment helps students to be motivated for learning and find learning interesting (Gillet, Berjot, Vallerand, & Amoura, 2012; León, Núñez, & Liew, 2015). This aspect has an influence on students' evaluation of their instructor's teaching effectiveness and student learning outcomes (Lizzio, Wilson, & Simons, 2002; Lovric et al., 2014). All these aspects of teaching can be controlled and facilitated by instructors. Nursing faculty can adopt more innovative teaching methods to arouse students' interest and motivation in learning the content, improve the course contents to become more practical and relevant to nursing students' needs, make a more systematic effort to create cooperative relationships, and offer more assistance as they strive to improve their teaching and the learning experiences of students (Papathanasiou, Tsaras, & Sarafis, 2014; Wolf, Bender, Beitz, Wieland, & Vito, 2004). The present study aimed to demonstrate how nursing programs may create a better learning experience for their students by identifying influential aspects of teaching so that instructors may better address their students' needs and strategically and systematically improve their teaching effectiveness to become excellent. Logistic regression analysis of SET data helps to identify which aspects of teaching are more influential on the overall experience of teaching as excellent. To date, very few studies have performed this analysis. Langbein and Snider (1999) tested the impact of SET ratings on the probability of retention and found that that poorly-rated courses negatively affect the retention of students in college. Teaching improvement using the systematic analysis of SET results may be a sensible and attainable effort faculty can make to achieve excellence in teaching and retain their students to accomplish successful graduation, rather than being passive and powerless because of variables (e.g., class, student, and instructor characteristics) that instructors usually have no control over. Our analysis identified five aspects that were not statistically influential on the overall excellence rating of the instructor's teaching: adequate syllabus, critical thinking skills, relevance to future profession, grading, and challenge. It is possible that the value of the syllabus is not significant to students when they evaluate the overall teaching quality of their instructors as excellent. Faculty view syllabi as a teaching and communication tool and take time to create a comprehensive and detailed syllabus, but students may not pay attention to items that faculty consider to be important information (Becker & Calhoon, 1999). Furthermore, there were unexpected findings that critical thinking skills and relevance of class content to future profession were not influential on students' rating of the instructors' overall teaching performance as excellent. These are highly emphasized and promoted in 21st century education, and health care professionals are especially required to have sophisticated critical thinking skills in order to address the dynamic and critical issues patients experience along with in-demand skills, such as problem-solving, collaboration, and interpersonal skills. Critical thinking skills are required to effectively care for different patient populations at different health priority levels. In fact, the importance of critical thinking in nursing practice
90
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
3
Odds Ratio (95%CI)
2.5 2 1.5 1 0.5 0 Q9. Motivation
Q5. Individual Assistance
Q3. Clear Presentation
Q6. Environment
Q4. Practical Application
Influential Aspects of Teaching Fig. 1. Relative importance of influential aspects on the overall rating as excellent.
has been reported, and various teaching strategies to promote critical thinking skills for nursing students have been introduced (Latif, Mohamed, Dahlan, & Mat Nor, 2016). In addition, critical thinking is a significant factor in nursing educational programs (Raymond, Profetto-McGrath, Myrick, & Strean, 2017). Nursing students might value other aspects of teaching as part of a positive learning experience, which might not necessarily fully reflect the main objectives of nursing programs. Another interesting finding is that grading practice did not affect the overall evaluation of teaching as excellent. There are mixed results about the relationship between grading and SET (Heckert, Latier, Ringwald-Burton, & Drazen, 2006). Grading has been found to influence students' evaluation of teaching effectiveness in disciplines, such as social work (Jirovec, Ramanathan, & Rosegrant-Alvarez, 1998) and public policy (Cho et al., 2014), but Griffin et al. (2014) reported a lack of correlation between grades and SET in religion classes and advised administrators to take caution not to misinterpret high grades and high SET as evidence of a cause and effect relationship between grades and SET. These inconsistent and unexpected results may be because of the nature of grading as a potential source of bias in higher education and may be related to unique perceptions students have about teaching and learning in different disciplines and students' main goal of enrolling in a higher education program, which necessitates the focus on learning practical applications rather than grades. Consistent with Griffin et al. (2014), the current finding that a clear grading practice tied to course objectives does not affect students' rating of their overall learning experience as excellent may be the result of students being unaware of the value of instructors' actions related to clarifying and linking grading to course objectives, which is a key component of a typical program evaluation report to demonstrate the quality of the program. In short, it is likely that students do not care about the same things faculty, administrators, and program accreditation evaluators care about. The current findings confirmed that numerical data in SET can enable instructors to clearly
see the aspects of teaching they need to focus on to meet the needs of their students in a timely manner and can be used as an effective tool by faculty in developing teaching excellence in nursing. It should be noted that student course evaluation is not the same as customer satisfaction. University instructors must be strategic and selective in how they reflect and implement the findings of the current research in their teaching practices because they have a comprehensive understanding of the mission and objectives of the program and courses and are committed to improve their pedagogic expertise in nursing education. Limitations and Implications The current paper investigated the aspects of teaching that influence the overall rating of an instructor's teaching as excellent. Although the results are promising, this study has several limitations, including the single-program nature of the sample and the sampling method. We used a sample in one nursing department, which would limit the generalizability of the study results. To generate more generalizable results, this SET study should be replicated in other nursing programs to investigate the levels of influence and patterns of various aspects of teaching that are valued by nursing students. Such investigation will certainly be challenging (Wolfer & Johnson, 2003) but is critical and timely to give information about more effective and preferred learning experiences of nursing students. Another critical requirement for future research is to obtain the perspectives of the instructors to discover the mismatches between teachers and students in aspects of teaching they value as excellence in teaching and delivering nursing knowledge and skills. Because of the anonymous nature of SET, external factors, such as student and course characteristics and instructor information were not obtained, but we are aware of the possibility that these factors might have influenced the overall rating of the instructors' teaching as excellent in the current study. However, the focus of the current study was on the controllable instructional variables that can be
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
constructively approached and managed by instructors, and the present results are consistent with previous findings on the power of motivation in student learning experiences. Nursing student evaluations are complex because of the overlap of learning and application in a variety of learning settings. Future research in nursing SET should make use of accessible characteristics of students, classes, and instructors and explore their relationships with multidimensional aspects of teaching and the overall rating of instructors as excellent in nursing programs (Annan et al., 2013). The dependent variable (i.e., the overall rating of instructors' teaching as excellent), was dichotomous (excellent or not), as the current study focused on examining the relative importance of the 10 aspects on the dependent variable. A logistic regression analysis was appropriate for the purpose of the study. The results of the analysis made it evident that simple comparisons of raw mean scores would not allow instructors to identify influential factors on the overall evaluation of their teaching as excellent. For example, the course syllabus is not an influential aspect of teaching on the overall rating as excellent, but this aspect received one of the highest mean scores. Nevertheless, focusing only on the two categories of strongly agree (5) or not (4, 3, 2, and 1) in the dependent variables may result in loss of information even though there is only a small number of ratings of 3, 2, or 1. In order to make full use of all the information, an ordinal logistic regression model may be considered. Conclusion The contribution of the present study can be found in its focus on influential yet controllable aspects of teaching that have a significant impact on the overall rating of instructors' teaching performance as excellent in a university nursing program. This study reported nursing students' unique focus on certain aspects of teaching when they evaluate whether their instructor's overall performance was excellent or not. Nursing students indicated that they placed different levels of importance on certain aspects of teaching, including motivation, individual assistance, clear presentation of concepts, learning environment, and practical application. When students assessed the overall excellence of their instructors' teaching, they were not concerned about the quality of a syllabus, relevance of the course content to their future profession, critical thinking skills, grading, and challenge in a course. Apparently, aspects of teaching that typically receive a great deal of faculty time and effort and are treasured by instructors as a way to communicate course expectations and objectives are not perceived at the same level by students. By being consumers of the results of SET who systematically analyze the results and prioritize the aspects of teaching to demonstrate excellence in teaching, faculty can provide students with better learning opportunities in a positive and motivating learning environment, adopt diverse teaching practices to emphasize various aspects of teaching that are critical for student learning and progress, and help students to be confident in the skills they learned so that they can effectively and successfully work in interprofessional health care settings and provide the highest quality of care to their future patients. Acknowledgments The authors would like to thank Regina Gordon and Adam Dircksen for their assistance with preparation of the aggregated data. References Agbetsiafa, D. (2010). Evaluating effective teaching in college level economics using student ratings of instruction: A factor analytic approach. Journal of College Teaching and Learning, 7(5), 57–66.
91
Al-Maamari, F. (2015). Response rate and teaching effectiveness in institutional student evaluation of teaching: A multiple linear regression study. Higher Education Studies, 5(6), 9–20. American Association of Colleges of Nursing (2017). Nursing faculty shortage fact sheet. Washington, DC: Rosseter, R [Retrieved from http://www.aacnnursing.org/ Portals/42/News/Factsheets/Faculty-Shortage-Factsheet-2017.pdf]. Annan, S. L., Tratnack, S., Rubenstein, C., Metzler-Sawin, E., & Hulton, L. (2013). An integrative review of student evaluations of teaching: Implications for evaluation of nursing faculty. Journal of Professional Nursing, 29(5), e10–e24. https://doi.org/10. 1016/j.profnurs.2013.06.004. Becker, A. H., & Calhoon, S. K. (1999). What introductory psychology students attend to on a course syllabus. Teaching of Psychology, 26(1), 6–11. Boysen, G. A. (2015). Uses and misuses of student evaluations of teaching: The interpretation of differences in teaching evaluation means irrespective of statistical information. Teaching of Psychology, 42(2), 109–118. https://doi.org/10.1177/ 0098628315569922. Braga, M., Paccagnella, M., & Pellizzari, M. (2014). Evaluating students' evaluations of professors. Economics of Education Review, 4171–4188. https://doi.org/10.1016/j. econedurev.2014.04.002. Campbell, H. E., Steiner, S., & Gerdes, K. (2005). Student evaluations of teaching. Journal of Public Affairs Education, 11(3), 211–231. Carbone, A., Ross, B., Phelan, L., Lindsay, K., Drew, S., Stoney, S., & Cottman, C. (2015). Course evaluation matters: Improving students' learning experiences with a peer-assisted teaching programme. Assessment & Evaluation in Higher Education, 40(2), 165–180. https://doi.org/10.1080/02602938.2014.895894. Cashin, W. E., & Downey, R. G. (1992). Using global student rating items for summative evaluation. Journal of Educational Psychology, 84(4), 563–572. Cho, J., Otani, K., & Kim, B. J. (2014). Differences in student evaluations of limited-term lecturers and full-time faculty. Journal on Excellence in College Teaching, 25(2), 5–24. Clayson, D. E., & Haley, D. A. (2011). Are students telling us the truth? A critical look at the student evaluation of teaching. Marketing Education Review, 21(2), 101–112. https://doi.org/10.2753/MER1052-8008210201. Divoky, J. J., & Rothermel, M. A. (1988). Student perceptions of the relative importance of dimensions of teaching performance across a type of class. Educational Research Quarterly, 12(3), 40–45. Dziuban, C., & Moskal, P. (2011). A course is a course is a course: Factor invariance in student evaluation of online, blended and face-to-face learning environments. The Internet and Higher Education, 14(4), 236–241. https://doi.org/10.1016/j.iheduc. 2011.05.003. Emery, C. R., Kramer, T. R., & Tian, R. G. (2003). Return to academic standards: A critique of student evaluations of teaching effectiveness. Quality Assurance in Education, 11(1), 37–46. Fraile, R., & Bosch-Morell, F. (2015). Considering teaching history and calculating confidence intervals in student evaluations of teaching quality. Higher Education, 70 (1), 55–72. https://doi.org/10.1007/s10734-014-9823-0. Gillet, N., Berjot, S., Vallerand, R. J., & Amoura, S. (2012). The role of autonomy support and motivation in the prediction of interest and dropout intentions in sport and education settings. Basic and Applied Social Psychology, 34(3), 278–286. https:// doi.org/10.1080/01973533.2012.674754. Griffin, A., & Cook, V. (2009). Acting on evaluation: Twelve tips from a national conference on student evaluations. Medical Teacher, 31, 101–104. Griffin, T. J., Hilton, J., III, Plummer, K., & Barret, D. (2014). Correlation between grade point averages and student evaluation of teaching scores: Taking a closer look. Assessment & Evaluation in Higher Education, 39(3), 339–348. https://doi.org/10. 1080/02602938.2013.831809. Girija, K. (2012). Effective clinical instructor– a step toward excellence in clinical teaching. International Journal of Nursing Education, 4(1), 25–27. Heckert, T. M., Latier, A., Ringwald-Burton, A., & Drazen, C. (2006). Relations among student effort, perceived class difficulty appropriateness, and student evaluations of teaching: Is it possible to “buy” better evaluations through lenient grading? College Student Journal, 40(3), 588–596. Hendry, G. D., Lyon, P. M., & Henderson-Smart, C. (2007). Teachers' approaches to teaching and responses to student evaluation in a problem-based medical program. Assessment & Evaluation in Higher Education, 32(2), 143–157. https://doi. org/10.1080/02602930600801894. Jirovec, R. L., Ramanathan, C. S., & Alvarez, A. R. (1998). Course evaluation: What are social work students telling us about teaching effectiveness? Journal of Social Work Education, 34(2), 229–236. Iqbal, I. (2013). Academics' resistance to summative peer review of teaching: Questionable rewards and the importance of student evaluations. Teaching in Higher Education, 18(5), 557–569. https://doi.org/10.1080/13562517.2013.764863. Johnson-Farmer, B., & Frenn, M. (2009). Teaching excellence: What great teachers teach us. Journal of Professional Nursing, 25(5), 267–272. https://doi.org/10.1016/ j.profnurs.2009.01.020. Johnson, M. D., Narayanan, A., & Sawaya, W. J. (2013). Effects of course and instructor characteristics on student evaluation of teaching across a college of engineering. Journal of Engineering Education, 2(102), 289–318. https://doi.org/10.1002/jee. 20013. Latif, R. A., Mohamed, R., Dahlan, A., & Mat Nor, M. Z. (2016). Concept mapping as a teaching tool on critical thinking skills and academic performance of diploma nursing students. Education in Medicine Journal, 8(1), 67–74. https://doi.org/10. 5959/eimj.v8i1.406. Langbein, L. I., & Snider, K. (1999). The impact of teaching on retention: Some quantitative evidence. Social Science Quarterly, 80(3), 457–472.
92
J. Cho et al. / Teaching and Learning in Nursing 13 (2018) 86–94
León, J., Núñez, J. L., & Liew, J. (2015). Self-determination and STEM education: Effects of autonomy, motivation, and self-regulated learning on high school math achievement. Learning & Individual Differences, 43, 156–163. https://doi.org/10. 1016/j.lindif. 2015.08.017. Lerret, S. M., & Frenn, M. (2011). Challenge with care: Reflections on teaching excellence. Journal of Professional Nursing, 27(6), 378–384. https://doi.org/10.1016/j. profnurs. 2011.04.014. Lizzio, A., Wilson, K., & Simons, R. (2002). University students' perceptions of the learning environment and academic outcomes: Implications for theory and practice. Studies in Higher Education, 27(1), 27–52. Lovric, R., Nrlic, N., Barac, I., Pluzaric, J., Puseljic, S., Berecki, I., & Radic, R. (2014). Specificities and differences in nursing students' perceptions of nursing clinical faculties' competencies. Journal of Professional Nursing, 30, 406–417. https://doi.org/ 10.1016/j.profnurs.2014.03.005. Matos-Diaz, H., & Ragan, J. F. (2010). Do student evaluations of teaching depend on the distribution of expected grade? Education Economics, 18(3), 317–330. https://doi. org/10.1080/09645290903109444. Morgan, H. K., Purkiss, J. A., Porter, A. C., Lypson, M. L., Santen, S. A., Christner, J. G., & Hammoud, M. M. (2016). Student evaluation of faculty physicians: Gender differences in teaching evaluations. Journal of Women's Health, 25(5), 453–456. https:// doi.org/10.1089/jwh.2015.5475. Morley, D. (2014). Assessing the reliability of student evaluations of teaching: Choosing the right coefficient. Assessment & Evaluation in Higher Education, 39(2), 127–139. https://doi.org/10.1080/02602938.2013.796508. Oermann, M. (2017). Student evaluations of teaching: There is more to course evaluations than student ratings. Nurse Educator, 42, 55–56. https://doi.org/10.1097/ NNE0000000000000366. Otani, K., Kim, B. J., & Cho, J. (2012). Student course evaluation (SET) in higher education: How to use student course evaluation more effectively and efficiently in public affairs education. Journal of Public Affairs Education, 18(3), 531–544. Papathanasiou, I., Tsaras, K., & Sarafis, P. (2014). Views and perceptions of nursing students on their clinical learning environment: Teaching and learning. Nurse Education Today, 34(1), 57–60. https://doi.org/10.1016/j.nedt.2013.02.007. Pintrich, P. R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95(4), 667–686. Ramsden, P. (2003). Learning to teach in higher education. London, UK: Taylor & Francis. Raymond, C., Profetto-McGrath, J., Myrick, F., & Strean, W. B. (2017). An integrative review of the concealed connection: Nurse educators' critical thinking. The Journal of
Nursing Education, 56(11), 648–654. https://doi.org/10.3928/0148483420171020-03. Setari, A. P., Lee, J., & Bradley, K. D. (2016). A psychometric approach to the validation of a student evaluation of teaching instrument. Studies in Educational Evaluation, 51, 77–87. https://doi.org/10.1016/j.stueduc.2016.09.006. Spiller, D., & Harris, T. (2013). Learning from evaluations: Probing the reality. Issues in Educational Research, 23(2), 258–268. Stehle, S., Spinath, B., & Kadmon, M. (2012). Measuring teaching effectiveness: Correspondence between students' evaluations of teaching and different measures of student learning. Research in Higher Education, 53(8), 888–904. https://doi.org/ 10.1007/s11162-012-9260-9. Surgenor, P. (2013). Obstacles and opportunities: Addressing the growing pains of summative student evaluation of teaching. Assessment & Evaluation in Higher Education, 38(3), 363–376. https://doi.org/10.1080/02602938.2011.635247. Tobin, R. R. (2017). Too early for physics? Effect of class meeting time on student evaluations of teaching in introductory physics. Physics Teacher, 55(5), 276–279. Tuazon, A. A. (2015). Comparison of core curriculum and distribution models of general education: Student evaluation of general education social science courses at the University of the Philippines Diliman. Philippine Social Sciences Review, 67(1), 17–32. Valiga, T. (2017). Do schools of nursing truly value excellence in teaching? Actions speaks louder than words. Journal of Nursing Education, 56(9), 519–520. https:// doi.org/10.3928/01484834-20170817-01. Wolf, Z. R., Bender, P. J., Beitz, J. M., Wieland, D. M., & Vito, K. O. (2004). Strengths and weaknesses of faculty teaching performance reported by undergraduate and graduate nursing students: A descriptive study. Journal of Professional Nursing, 20(2), 118–128. https://doi.org/10.1016/j.profnurs.2004.03.003. Wolfer, T., & Johnson, M. M. (2003). Re-evaluating student evaluation of teaching: The teaching evaluation form. Journal of Social Work Education, 39(1), 111–121. Wong, W. Y., & Moni, K. (2014). Teachers' perceptions of and responses to student evaluation of teaching: Purposes and uses in clinical education. Assessment & Evaluation in Higher Education, 39(4), 397–411. https://doi.org/10.1080/ 02602938.2013.844222. Wood, M., & Su, F. (2017). What makes an excellent lecturer? Academics' perspectives on the discourse of ‘teaching excellence’ in higher education. Teaching in Higher Education, 22(4), 451–466. https://doi.org/10.1080/13562517.2017.1301911. Zabaleta, F. (2007). The use and misuse of student evaluations of teaching. Teaching in Higher Education, 12(1), 55–76.
Our reference: TELN 531
P-authorquery-v11
AUTHOR QUERY FORM Journal: TELN
Please e-mail your responses and any corrections to:
Article Number: 531
E-mail: [email protected]
Dear Author, Please check your proof carefully and mark all corrections at the appropriate place in the proof (e.g., by using on-screen annotation in the PDF file) or compile them in a separate list. Note: if you opt to annotate the file with software other than Adobe Reader then please also highlight the appropriate place in the PDF file. To ensure fast publication of your paper please return your corrections within 48 hours. For correction or revision of any artwork, please consult http://www.elsevier.com/artworkinstructions. We were unable to process your file(s) fully electronically and have proceeded by Scanning (parts of) your article
Rekeying (parts of) your article
Scanning the artwork
Any queries or remarks that have arisen during the processing of your manuscript are listed below and highlighted by flags in the proof. Click on the ‘Q’ link to go to the location in the proof.
Location in article
Query / Remark: click on the Q link to go Please insert your reply or correction at the corresponding line in the proof
Q1
The author names have been tagged as given names and surnames (surnames are highlighted in teal color). Please confirm if they have been identified correctly.
Q2
Please provide city and postal codes for all affiliations.
Q3
Ref. "Frail & Bosch-Morell, 2015" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q4
The citation "Carbone, 2015" has been changed to "Carbone et al., 2015" to match the author name/ date in the reference list. Please check if the change is fine in this occurrence and modify the subsequent occurrences, if necessary.
Q5
Ref. "Morgan, 2015" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q6
Ref. "Braga, 2014" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q7
Ref. "Dziuban et al., 2011" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q8
Ref. "Johnson, Narayanan, & Sawaya, 2013" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q9
Ref. "Iqbal, 2012" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q10
Ref. "Johnson-Farmer & Frenn, 2009" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Page 1 of 2
Our reference: TELN 531
P-authorquery-v11
Q11
Ref. "Girija, 2012" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q12
"CI" was inserted here to mean as "confidence interval." Please check if correct.
Q13
Refs. "Cashin et al., 1992", "Wood et al.’s (2017)" are cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q14
Ref. "Lanbein and Snider (1999)" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q15
Ref. "Jirovec, Ramanathan, & Rosegrant-Alvarez, 1998" is cited in text but not provided in the reference list. Please provide details in the list or delete the citation from the text.
Q16
Uncited references: This section comprises references that occur in the reference list but not in the body of the text. Please position each reference in the text or, alternatively, delete it. Thank you.
Q17
Please supply last page. Please check this box if you have no corrections to make to the PDF file.
□
Thank you for your assistance.
Page 2 of 2