- Email: [email protected]
Assessing metacognition in the classroom: Student help-seeking behavior
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Research Note
Assessing metacognition in the classroom: Student help-seeking behavior Youn Chu, Shannon Palmer, Adam M. Persky
⁎
Division of Pharmacotherapy and Experimental Therapeutics, Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, 2312 Kerr Hall, CB#7569, Chapel Hill, NC 27599, United States
A R T IC LE I N F O
ABS TRA CT
Keywords: Academic help-seeking behavior Metacognition Capstone Pharmacy students Social desirability
Introduction: The study's purpose was to develop an assessment of students’ metacognitive monitoring of help-seeking behavior. Methods: This study piloted an assessment of help-seeking behavior in first-year student pharmacists to answer two questions: (1) Does help-seeking behavior depend on how familiar students are with the content? and (2) When students ask for help, does their performance and metacognition differ from when they do not seek help? As part of their year-end capstone, students answered drug information questions. The drugs within these questions were chosen based on the level of emphasis during the first-year curriculum (i.e. more familiar or less familiar). For each question, students rated their confidence level for their answer's correctness and marked whether they would ask their preceptor for help. Bias scores were calculated under conditions of familiarity based on level of emphasis (more familiar vs. less familiar) and help-seeking (asked for help and did not ask for help). Results: Students performed better on more familiar material (d = 1.2), with a small difference in confidence on more familiar material (d = 0.2). When students asked for help, they scored lower (d = −2.2) and reported lower confidence (d = −3.7). Students were more likely to ask for help from their preceptors on more familiar content than less familiar (odds ratio = 1.25) and less likely to ask for help when they were overconfident (odds ratio = 0.18). Conclusions: Overall, students were more overconfident for less familiar material and were less likely to ask for help.
Introduction Imagine the following situation: Elisa is on her hospital introductory pharmacy practice experience (IPPE) and is asked to recommend an appropriate dose of a medication by the rounding team. The answer comes to her mind quickly and she states the dose with confidence. However, the dose that she recommends would have significant adverse effects. This medical error is completely preventable. In this case, Elisa associated how quickly an answer came to mind with “correctness” of the answer which may be the result of poor metacognition.1–3 Instead, she should have asked her preceptor or looked up the dosing information in the appropriate resources. In simple terms, Elisa should have been better aware of what she knew (or did not know), how well she knew it, and how that may relate to her confidence in her knowledge. She should have asked for help. Being cognizant of what one does not know is an important skill in a learner's development.4 Student pharmacists and other health profession students must utilize help-seeking skills
⁎
Corresponding author. E-mail address: [email protected] (A.M. Persky).
https://doi.org/10.1016/j.cptl.2018.08.011 Received 20 February 2018; Received in revised form 19 June 2018; Accepted 3 August 2018 1877-1297/ © 2018 Elsevier Inc. All rights reserved.
Please cite this article as: Chu, Y., Currents in Pharmacy Teaching and Learning (2018), https://doi.org/10.1016/j.cptl.2018.08.011
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
during clinical experiences to fill their gaps in knowledge. Because there are few studies investigating how to assess students’ competency in asking for help, the purpose of this study is to pilot a metacognitive assessment aimed at characterizing student pharmacists’ likelihood to seek help in preparation for early immersion experience. Help-seeking behavior is a type of metacognitive monitoring or the process a learner uses to evaluate how well they have learned material or performed a task. In this case, the output for the monitoring is help-seeking, a form of metacognitive control, the use of metacognitive monitoring to guide one's future behavior. Help-seeking behavior can be defined as “an achievement behavior involving the search for and employment of a strategy to obtain success”.5 A theoretical framework for help-seeking has been established and consists of five steps: (1) become aware of need for help, (2) decide to seek help, (3) identify potential helper(s), (4) use strategies to elicit help, and (5) evaluate help-seeking episode.6,7 The critical first step of this model challenges the learner to assess task difficulty, monitor their task progress, and evaluate their own comprehension. That is, learners need to use metacognitive monitoring; they need to be aware and understand their own thought process. If learners cannot accurately monitor what they have learned, they will struggle with decisions on what material needs to be studied, for how long, and when to ask for assistance.8–10 In other words, if learners do not have appropriate metacognitive monitoring skills, they will struggle with metacognitive control and acting in their best self-interests (i.e. self-regulated learning). The ability to accurately judge task progress and comprehension requires metacognitive skills. Students must become efficient in this first step of the help-seeking process to develop into selfregulated learners that act in their own, long-term best interest.11 As such, we focused efforts to assess the initial two steps in the help-seeking process, becoming aware for the need of help and deciding to seek help. Studies have found a relationship between help-seeking behavior and academic success. Ryan et al.12 found that with learners as young as sixth grade, adaptive help-seeking behavior has been associated with better outcomes in school. In student pharmacists, academic help-seeking behavior was associated with greater academic competence and more positive faculty/student relationships.13 Due to the positive impact help-seeking behavior appears to have in the classroom, we wanted to explore pharmacy students’ metacognition in relation to help-seeking behavior. That is, are student pharmacists aware enough of their knowledge and skills to know when to ask for help? However, studying help-seeking behavior is complex, impacted by instructional methods, environment, and learners’ self-concepts.14 To date, there has been very little work on the metacognitive monitoring involved in help-seeking. In this current study, we used a written assessment to help evaluate student's metacognitive monitoring of help-seeking behavior. Data on assessing help-seeking behavior through assessment is very limited, if not completely absent. However, there is a plethora of literature on metacognitive monitoring using judgments of learning.15–17 A judgment of learning (JOL) is defined as one's perceptions of one's own ability. This current study and its methods are an extension of this literature but applied to a follow-up action, helpseeking. Within the JOL literature, comprehension monitoring is linked to the formation of a JOL and, consequentially, control of selfregulated learning.17,18 As an example, a JOL can evaluate how well one has understood information or performed a task; this can include scoring one's own confidence after being asked a question.17,18 In addition, tutoring systems have been used to look at using feedback to improve students’ help-seeking skills, particularly in younger students.7 These systems provide hints as students proceed through the program. These hints help develop metacognitive behaviors and improved help-seeking skills by giving the learner cues to change their behavior.7 For example, one system, aimed at high school geometry instruction, used a cognitive model to track students in their various approaches through geometry problems, estimating their knowledge level, and providing domain-level feedback and on-demand hints regarding the steps of these problems. This tutoring system indicated that help-seeking behavior can be trained within a student population even if it is not obviously stated. Although we are not assessing transferability of the implicit learning in this study, we hope that the same implicit effects seen within the tutoring system literature can be seen with this written assessment as it might impact performance in the experiential part of the curriculum. During IPPEs and advanced pharmacy practice experiences (APPEs), student pharmacists are given tasks and problems they have not covered in school, yet they are expected to use available resources, including their preceptors, to arrive at the correct answer. We expect students will ask for help when they have less background knowledge (less familiar material) than material they have prior exposure to within the curriculum. Thus, it is important to assess students’ help-seeking behavior under conditions of more familiar and less familiar information. Stavrianopoulos found students with better metacognitive monitoring abilities reviewed material they had previously indicated as unfamiliar.19 This supports the work about error reduction proposed by Metcalfe et al.20 that students will study material with the biggest difference between their JOL (how well they think they know a topic) and their actual performance. By looking at material that students had varying levels of familiarity with (i.e. familiar vs. less familiar material), we examined the impact on students’ metacognition and help-seeking behavior. Besides the issue of familiarity, we also expect that the learners with higher degrees of overconfidence (predicted score greater than actual score) are less likely to ask for help compared to students with stronger degrees of underconfidence (predicted score less than actual score). In this study, we measured calibration or metacognitive accuracy through the difference in confidence judgments and performance. The accuracy of metacognitive monitoring can be impacted by various factors including problem difficulty, background (or prior) knowledge (i.e. more familiar or less familiar information) and desired performance level.21–23 Bias is the difference between the confidence judgment and actual performance and includes both directionality (i.e. under- or overconfident) and magnitude of that directionality; absolute bias captures the magnitude or accuracy only.24,25 In prior research, overconfidence has been associated with early termination of study and poorer retention of material.18,19 Therefore, we may expect that students with more overconfidence would be less likely to ask for help.26,27 Taken together, this study piloted an assessment of academic help-seeking behavior of student pharmacists at the end of their first year and getting ready to transition to their first experiential setting. There were two primary questions: (1) Does help-seeking behavior depend on content familiarity and (2) When students ask for help, do their performance and confidence differ from when they do not ask for help? Several subquestions will be addressed that examine metacognitive monitoring for higher or lower 2
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
performing students and to identify factors that may predict help-seeking behavior. The overall goal is to develop a process to assess students metacognitive monitoring of help-seeking. Methods The help-seeking behavior assessment was within the find-and-apply assessment of the first-year capstone at the University of North Carolina Eshelman School of Pharmacy. The goal of the capstone assessment was to provide feedback to students, preceptors, and the school. As part of a new curriculum, we developed a capstone assessment administered after the first year to help prepare students for their first early immersion experience (e.g., IPPE). The capstone is described elsewhere and included four components: (1) a closed book knowledge assessment, (2) an open resource find-and-apply assessment, (3) an assessment of interpersonal skills, and (4) a debriefing session.28 The find-and-apply assessment is an open-resource assessment of students’ ability to answer drug information questions by finding resources and applying their knowledge and interpretation of those resources. Students were given four hours to complete this assessment. Questions were multiple-choice with a single best correct answer and ranged from identifying key adverse effects and counseling a patient to drug substitutions and identifying the severity of drug interactions. The questions were developed to stretch the students’ ability by not focusing on recallable facts but focusing on utilizing drug information resources (e.g., PubMed, Micromedex, UpToDate) to answer unfamiliar questions. Students could use any resource on their computer or internet including electronic library resources. This section contained 73 questions with 48 questions related to the current study. Of these 48 questions, 16 were drug information questions and 32 questions were used to assess help-seeking behavior. For the 16 drug information questions, assessed drugs were selected based on the school's curriculum map. Eight drugs were selected because they were discussed in multiple courses with some depth including being assessed. This was deemed the more familiar material. Another eight drugs were selected because they were either used as examples only or were not discussed within first-year courses. These were deemed the less familiar drugs. While these less familiar drugs may be more variable in familiarity because of students work or clinical experience outside of class, we were certain based on course work they were not emphasized during formal instruction and should still be unfamiliar enough for purposes of this study. The 16 drug information questions were graded as correct or incorrect with no partial credit. For the 32 questions used to assess help-seeking behavior, after answering the drug information question, students were asked to make a confidence judgment (continuous scale of 0–4 where 0 = no confidence and 4 = high confidence or 100% confident) on the correctness of their answer. They were then asked if they would ask their preceptor for help on that question (yes or no). Appendix 1 has an example of this question format. This study assessed students help-seeking relative to their judgment accuracy (how well a person’ s judgments are related to target performance). The absolute accuracy is the magnitude of the effect, because it reflects the absolute match between judgment magnitude and target performance.24 Relative accuracy is the degree to which the judgments discriminate between different levels of performance across items.24 These two measures are independent as a student can have good absolute accuracy but poor relative accuracy, and vice versa.24,25 While there are various methods to calculate absolute and relative accuracy, we used bias, absolute bias, and the correlation coefficient, gamma.24 For each group of questions (more familiar vs. less familiar; ask preceptor or not ask preceptor), scores were averaged as percent correct (e.g., 60% correct). Confidence judgments were also averaged and converted to a percentage (e.g., 70% confident). Bias score was calculated by the difference in confidence (as a percent) and correctness (as a percent) (bias = confidence − score). This is a measure of overconfidence (confidence greater than actual score) or underconfidence (confidence less than actual score). Negative values of bias are considered underconfident and positive values are considered overconfident. For this study we also defined a third category called “well-calibrated” for confidence judgment within 20% of performance (e.g., the average score was 70% but average confidence was 60–80%). In addition, an absolute bias score was calculated (absolute bias = absolute value of bias) which indicates the magnitude of the difference between the confidence judgment and performance regardless of directionality. Relative accuracy for each individual was computed as a gamma correlation between the confidence judgments and actual performance.29 This was a within-subjects design because students answered both familiar and less familiar questions and asked or did not ask for help. As such, a paired t-test was used to determine differences between familiarity conditions (more/less) and asking for preceptor help (yes/no). Logistic regression was used to determine potential predictors of help-seeking behavior. This was performed with the dependent variable being asking for help (yes/no) and the independent variables being the average question score (continuous variable), average question confidence (continuous variable), familiarity of the topic (more/less), and bias (well-calibrated, underconfident, or overconfident). All variables were entered together. An odds ratio (OR) was calculated when appropriate using chisquare and a Cohen's d was used to determine effect sizes. For Cohen's d, a d < 0.4 was considered small, 0.4 < d < 0.7 was considered medium, and d > 0.7 was considered large.30 Finally, students were placed into tertiles based on examination performance and comparisons were made on performance and metacognitive metrics using an ANOVA with Tukey post-hoc test. Statistical analyses were performed utilizing SPSS31 and significance was set at p < 0.05. The study was deemed exempt by the university's institutional review board. Results Of the 143 first-year student pharmacists who participated in the study, 100 students had complete data sets; the excluded students did not answer one or more questions of confidence or asking a preceptor. The first question we sought to answer was “Does help-seeking behavior depend on content familiarity?”. To answer this question, 3
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
Fig. 1. Performance and confidence based on familiarity. Data presented as mean percent and standard deviation. Students’ scores were higher for more familiar content but a small difference in confidence.
we first explored differences in performance and confidence based on content familiarity. For more familiar material compared to less familiar material, students performed better (d = 1.2) and had higher confidence (d = 0.22), although not the same degree (Fig. 1). Overall, for familiar content, students had better relative accuracy (gamma = 0.62 vs 0.37) than less familiar content. Students were underconfident on the more familiar material than on the less familiar material (d = 0.87, p < 0.001). There was no difference in accuracy of judgment (i.e., absolute bias, extent of difference between confidence and score) between the two conditions. Finally, when faced with more familiar content, students were more likely to ask a preceptor for help [OR 1.25 (1.01, 1.56), p < 0.05]. The second question asked “When students ask for help, do their performance and confidence differ from when they do not ask for help?”. Only five students would not ask their preceptor for help on any question. On average, students would ask a preceptor for help on 71% of the questions (11 ± 3 questions, range 1–16). For questions which students would ask a preceptor for help, students scored lower (d = −2.2) and were less confident (d = −3.7) (Fig. 2). Overall, students’ relative accuracy was better when they asked the preceptor for help (gamma = 0.54 vs. 0.28). Students who asked for help on the question were more underconfident (d = 0.61, p < 0.001) and less accurate (d = 0.43, p = 0.005). The first two questions raised a third: “What factors may predict the desire to ask for help?” We performed an exploratory logistic regression with the dependent variable of asking a preceptor for help (yes or no). The question score, confidence level, topic familiarity, and confidence category (calibrated, underconfident or overconfident) were entered into the model. The model explained 85% of the variability (Nagelkerke R2) (Table 1). Students were less likely to ask for help with less familiar content than more familiar content. As performance and confidence increased, the odds of students asking for help decreased. For a 1% increase in score (e.g., 50% to 51%), the odds a student would ask for help decreased by 4% when other factors are held constant. For a 1% increase in confidence (70% to 71%), the odds of asking for help decreased by 14% when other factors are held constant. Finally, compared to well-calibrated students (confidence within ± 10% points of performance), overconfident students were less likely to ask for help but underconfident students were equally likely to ask for help (Table 1, Fig. 3). This suggests more familiar material influenced students’
Fig. 2. Performance and confidence for whether students asked a preceptor for help. Data presented as mean percent and standard deviation. Students’ scores and confidence were higher when asking for help than when not asking for help. 4
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
Table 1 Summary of logistic regression. Metric
Standard
Score (%) Confidence (%) Less familiar Overconfident Underconfident
– – More familiar Well-Calibrated
−.046 −.15 −1.6 −1.7 −1.2
Odds ratio
95% Confidence interval
0.96a 0.86a 0.20a 0.18a 0.30
0.92, 0.99 0.82, 0.90 0.067, 0.59 0.036, 0.93 0.047, 1.2
a
p < 0.05. Scores = performance on the questions; Confidence = confidence students had on their answers; Less familiar = content that was less familiar to students because it was less emphasized in the curriculum; Overconfident = determined by confidence greater than score; Underconfidence = confidence less than score; Well-calibrated = when the confidence and scores were closely matched.
Fig. 3. Confidence versus performance and zones of help seeking. Zone I = Underconfidence; Zone 2 = Well-calibrated; Zone III = Overconfidence. Open-circles = average score and confidence on questions students asked for help; Closed circles = average score and confidence for questions students did not ask for help on.
likelihood to ask for help more than their level of confidence or how well they performed. We further explored data based on student performance because students who perform lower also tend to be less metacognitively aware.32,33 As such, our final set of questions probed if better performing students behave differently than lower performing students. Students were separated into tertiles based on performance on the 16 content-related capstone questions. When examining the impact of familiarity on help-seeking, the lowest tertile had lower confidence compared to the top tertile. The lowest performing students also had higher bias scores (i.e. more positive/less negative) and lower absolute bias compared to the top two tertiles (Fig. 4A, Table 2). For the less familiar material, only the lowest tertile had higher bias compared to higher and middle performing students (Fig. 4B, Table 2). There were no differences with respect to confidence ratings between the tertiles. We were also interested in learning if lower performing students had differential metacognitive monitoring when asking for help. The lowest performing students had lower confidence, higher bias scores, and lower absolute bias scores when they asked for help (Fig. 4C, Table 2). There were no differences in metacognitive monitoring for questions on which students did not ask for help (Fig. 4D, Table 2).
Discussion The purpose of this study was to examine a novel way to assess the metacognitive monitoring of help-seeking behavior. As expected, students performed better on more familiar material than less familiar material and reported slightly higher confidence on more familiar material. Surprisingly, students were less likely to ask for help on less familiar than more familiar material which may support the “uninformed and unaware” phenomena.34 Students were less likely to ask a preceptor for help if they were overconfident, especially for unfamiliar material. This may reflect the Dunning-Kruger effect which is a cognitive bias wherein individuals hold an overly optimistic and miscalibrated view about their abilities. Low knowledge individuals overestimate their ability (i.e. overconfident) and high knowledge individuals will underestimate their abilities (i.e. underconfident). We found that students were more likely to be underconfident on more familiar material (i.e. skilled and aware). The more familiar material is subject to memory error, which may reduce confidence level.35 That is, students were aware they once knew the material but are unaware of their current knowledge. Since the familiar drugs were taught and 5
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
Fig. 4. Performance and confidence based on tertiles of performance. (A) More familiar material; (B) Less familiar material; (C) When students would ask a preceptor for help; (D) When students would not ask a preceptor for help. The bottom tertile generally had differences in bias and absolute bias compared to the upper two tertiles. Table 2 Summary of parameters when students were asked about more familiar or less familiar and asked for help content based on performance on the questions. Condition
Tertile
Score
Confidence
Bias
Absolute Bias
More familiar
Top (n = 27) Middle (n = 30) Bottom (n = 43) Top (n = 27) Middle (n = 29) Bottom (n = 40) Top (n = 27) Middle (n = 30) Bottom (n = 43) Top (n = 27) Middle (n = 29) Bottom (n = 40)
84 77 65 72 60 44 70 63 44 96 88 86
64 58 57 60 56 54 50 46 43 91 89 87
−20 (13) −20 (11) −8 (15)a,b −12 (17) −4 (18) 10 (15)a,b −20 (17) −17 (14) −0.8 (13)a,b −5.2 (12) 0.3 (17) 1.2 (17)
21 (12) 20 (10) 13 (12)a,b 16 (13) 15 (12) 14 (12) 22 (14) 18 (12) 10 (8.3)a,b 9.3 (8.8) 12 (12) 12 (12)
Less familiar
Asked Preceptor
Did not ask preceptor
(9.0) (7.7) (12)a,b (8.1) (7.7)a (13)a,b (9.0) (11)a (9.7)a,b (7.9) (16) (17)a
a
(12) (12) (11)a (13) (13) (14) (13) (11) (11) (8.9) (11) (14)
p < 0.05 vs. Top Tertile; bp < 0.05 vs. Middle Tertile. Tertile: based on student performance; Bias = difference in confidence and score; Absolute bias = absolute value of bias. 6
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
applied in various courses at different times, they are more likely to be influenced by both proactive and retroactive interference. Conversely, for less familiar drugs, students had little experience with the drugs and assumed they could accurately find and utilize information (i.e. unskilled and unaware). Less familiar material has less chance for memory error because students had very limited exposure to these drugs, therefore, they reported more accurate confidence scores matching the performance score. Ultimately the overconfidence can be explained with how students use cues to predict how well they understand the material. In our early example, Elisa used a cue of how fast the answer came to mind as a cue to correctness of her answer. Overconfident JOLs (i.e. confidence judgments) are often explained by the cue utilization approach which states that for a learner to monitor their comprehension, they utilize cues that they view as predictive on how well they comprehend. That is, the learner has some internal check and balance system. However, research on cue utilization indicates that learners often have low comprehension monitoring skills because they frequently utilize cues that do not possess a high degree of predictive validity.36 In other words, their internal cues do not predict their actual comprehension. Therefore, it is possible, the ease at which learners found or recalled information suggests to them their answer is correct when in fact, it may not be. This ease of fluency has been associated with higher confidence judgments but lower or no differences in performance.21,37,38 In contrast, deeply comprehended information assumes the information has been sufficiently integrated into the learners’ prior knowledge and it is likely that learners are able to recall it. What this means is, we assume that the amount of recallable information often exceeds the amount of well comprehended information and, when using memory-based cues for comprehension monitoring, it can lead to overconfidence. When seeking help, this study found that familiarity was a strong predictor, more so than performance, confidence, or calibration. Students were more likely to ask for help on more familiar material than less familiar material, which contradicts our initial expectation but in hindsight may make sense. On one hand, if students had appropriate metacognitive accuracy, they would know when they encountered a new situation and ask for help. However, because students had knowledge of the topics, they were more aware of their limits and would ask for help. For less familiar material, students had little background knowledge which may lead to being unaware of the need for help. In addition, students may avoid asking their preceptor questions for which they have no or little baseline information, a form of social desirability.39 In this case, the hope is students would look up the drug information prior to approaching their preceptors for confirmation or clarification. For example, learners may not ask for help out of fear that they will receive less credit for a successful outcome or that the teacher or their peers will view them as incompetent.6,30 This may explain why students asked for help on over 70% of the questions, they may feel that is the socially acceptable response and do not want to appear incompetent.6,40 To put this study back into instructional context, this part of the capstone was designed to help provide feedback to students moving forward into their first clinical experiences. During the debrief session of the capstone, the preliminary data for the study was shown and discussed, especially about the less familiar content and being overconfident. The goal was to help bring awareness of student biases and discuss when it is appropriate to ask for assistance. A secondary goal was to reinforce the help-seeking behavior. As noted in the help-seeking literature, there is an implicit learning that occurs with asking for help.7 The question arises, “Does asking for help on a paper and pen assessment translate to actual help-seeking behavior?”. The answer is potentially with support from two frameworks. The behavioral consistency theory states that past behavior is the best predictor of future behavior. If asking for help in this assessment is an example of current behavior, then that allows prediction of future behavior within experiential training.41,42 The second framework is the implicit trait policies (ITPs) which proposes that individuals develop beliefs about the effectiveness of different behavior. That is, in each situation, individuals weigh the costs and benefits associated with expressing certain traits (i.e., asking for help) in context to an individuals’ inherent tendencies or traits, and dependent on the job level, specific job knowledge.43,44 For example, a pharmacist dealing with a sensitive situation may have to make a judgment about the utility (i.e. costs/benefits) of demonstrating empathy and agreeableness as a more successful strategy than acting brusquely or being disagreeable even if the pharmacist's preference tends towards being generally disagreeable and/or empathetic. There are several limitations to the current study. First, familiarity was based on the extent the drugs were discussed within the curriculum based on the school's curriculum map. It is difficult to quantify what justifies familiarity and it does not include exposure students may have outside of school through extra- and co-curriculars. However, based on performance, familiar material scored higher thus potentially confirming its familiarity to students. This also was confirmed by asking students to provide a rating of familiarity. Due to metacognitive bias. that students view of familiarity may not be the same as the expectations of the instructors’, we did not use the student ranking of familiarity. This ranking was there for confirmation purposes only, not to classify content. The second area is that help-seeking questions could be biased because of social desirability and it may be helpful in the future to have some measure of social desirability as an explanatory factor or maybe limit the opportunities for help-seeking to help assign more value to the strategy (i.e. make it a limited resource). The third limitation is students indicated they would ask a preceptor for help but there was no measure of them seeking help. However, there are at least two theories stated earlier that suggest students may be likely to ask for help based on the paper-and-pen assessment There are a couple of instructional implications based on the results of this study. The first implication is preparing health profession students to become clinicians involves cultivating both knowledge and behavioral skills. It is crucial for instructors and administrators to develop tools to measure student progress and ability across categories. This study provides an example of how assessments may be used to help measure students’ metacognition. These types of assessments provide a potential way to measure student growth over time. Schools of pharmacy could use help-seeking assessments pre- and post- immersion experiences to help measure if improvement occurred in this skill over the course of the experience. This could help demonstrate if students’ judgment of when to ask for help improved after seeing clinical decisions made in real-word settings. The second implication could be helping learners utilize confidence ratings on their own so they can avoid the ramifications of overconfidence and being “unskilled and unaware.” These confidence judgments tend to be more impactful if they are delayed versus made immediately.45,46 Regardless, these 7
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
judgments may include learners keeping a record of topics, rating their confidence on those topics, and, for new topics, checking their knowledge before trying to apply that knowledge. This would provide another method for students to visualize if they were overconfident or underconfident with responses. It should also be noted that even though students may have high levels of confidence, it does not mean they are going to make appropriate decisions regarding help-seeking. In this study, students had high levels of confidence but when comparing their confidence to their performance, their performance was greater than their confidence (i.e., overconfident). It was these students that would be more likely to seek help. Conclusions This study was successful in describing the metacognitive monitoring of help-seeking behavior. The study captures theoretically what students would do but it may be confounded by some aspects of social desirability. Regardless, follow-up studies may focus on the differences investigating further when students seek help and potential differences between seeking help from a preceptor versus a teacher, peer, or mentor. Conflict of interest None. Disclosures None. Acknowledgments The authors would like to thank the team that helped develop the capstone, especially: Jessica Greene and Kathryn Fuller who were instrumental in the Find-and-Apply assessment development. The authors also thank Sabrina Dunham for her review of the manuscript. Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.cptl.2018.08.011. Appendix 1. Sample drug information question KF is a 45-year old female with heartburn and indigestion. She was just started on prednisone 7.5 mg PO daily by her rheumatologist to help control rheumatoid arthritis (RA) disease activity. At her rheumatology appointment, her RAPID-3 score was 6. Past medical history of RA and type-1 diabetes. Meds
• Levothyroxine 100 mcg daily • Methotrexate 15 mg once weekly with folic acid 3 mg once daily • Acetaminophen 650 mg every 8 hours PRN • Celecoxib 100 mg twice daily • Calcium citrate 1500 mg/day • Multivitamin once daily • Insulin glargine 22 units once daily • Insulin lispro 6 units with breakfast, 7 units with lunch and 9 units with dinner • Prednisone 7.5 mg once daily X 1 week then 5 mg once daily X 1 week Allergies and ADRs: sulfonamide antibiotics (Rash); azithromycin (abdominal pain); amoxicillin (diarrhea) Blood Glucose log
Monday Tuesday Wednesday Thursday Friday
Before Breakfast
Before Lunch
128 103 87 103 74
127 78 72 90 125
Low BG symptoms
55 (3:46 p.m.) 60 (4:52 p.m.) 58 (4:07 p.m.) 8
Before Dinner
Bedtime
72 195 88 192 201
121 102 105 101 97
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
Saturday Sunday
88 83
98 88
62 (4:29 p.m.)
93 188
96 122
You are asked to counsel KF on the use of her prednisone. What would be the top priorities to counsel her on? A Water retention and development of a “moon face” B Elevated morning glucose and vivid dreams C Risk of osteoporosis and flu-like symptoms For the question you just answered, what is your level of confidence that your answer is correct? A B C D E
0 – no confidence 1 2 3 4 – high confidence For the question above, would you ask your preceptor for assistance?
A Yes B No PO = by mouth; PRN = as needed.
References 1. Benjamin AS, Bjork RA, Schwartz BL. The mismeasure of memory: when retrieval fluency is misleading as a metamnemonic index. J Exp Psychol Gen. 1998;127(1):55–68. 2. Serra MJ, Dunlosky J. Does retrieval fluency contribute to the underconfidence-with-practice effect? J Exp Psychol Learn Mem Cogn. 2005;31(6):1258–1266. 3. Koriat A, Ma'ayan H. The effects of encoding fluency and retrieval fluency on judgments of learning. J Mem Lang. 2005;52(4):478–492. 4. McLaughlin JE, Bush AA, Rodgers PT, et al. Exploring the requisite skills and competencies of pharmacists needed for success in an evolving health care environment. Am J Pharm Educ. 2017;81(6) https://doi.org/10.5688/ajpe816116. 5. Ames R, Lau S. An attributional analysis of student help-seeking in academic settings. J Educ Psychol. 1982;74(3):414–423. 6. Gall SN. Help-seeking: an understudied problem-solving skill in children. Dev Rev. 1981;1(3):224–246. 7. Roll I, Aleven V, McLaren BM, Koedinger KR. Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learn Instr. 2011;21(2):267–280. 8. de Bruin ABH, Dunlosky J, Cavalcanti RB. Monitoring and regulation of learning in medical education: the need for predictive cues. Med Educ. 2017;51(6):575–584. 9. Soderstrom NC, Clark CT, Halamish V, Bjork EL. Judgments of learning as memory modifiers. J Exp Psychol Learn Mem Cogn. 2015;41(2):553–558. 10. Townsend CL, Heit E. Judgments of learning and improvement. Mem Cogn. 2011;39(2):204–216. 11. Puustinen M. Help-seeking behavior in a problem-solving situation: development of self-regulation. Eur J Psychol Educ. 1998;13(2):271–282. 12. Ryan AM, Shin H. Help-seeking tendencies during early adolescence: an examination of motivational correlates and consequences for achievement. Learn Instr. 2011;21(2):247–256. 13. Payakachat N, Gubbins PO, Ragland D, et al. Academic help-seeking behavior among student pharmacists. Am J Pharm Educ. 2013;77(1) https://doi.org/10.5688/ ajpe7717. 14. Strategic Help Seeking: Implications for Learning and Teaching. St. Louis, MO: Routledge; 2012. 15. Handbook of Metamemory and Memory. New York, NY: Psychology Press, Taylor & Francis Group, LLC; 2008. 16. Dunlosky J, Metcalfe J. Metacognition. Los Angeles, CA: SAGE Publications, Inc.; 2008. 17. Metcalfe J. Metacognitive judgments and control of study. Curr Dir Psychol Sci. 2009;18(3):159–163. 18. Koriat A, Goldsmith M. Monitoring and control processes in the strategic regulation of memory accuracy. Psychol Rev. 1996;103(3):490–517. 19. Stavrianopoulos K. Adolescents' metacognitive knowledge monitoring and academic help seeking: the role of motivation orientation. Coll Stud J. 2007;41(2):444–453. 20. Metcalfe J, Kornell N. A region of proximal learning model of study time allocation. J Mem Lang. 2005;52(4):463–477. 21. Pressley M, Ghatala ES. Delusions about performance on multiple-choice comprehension tests. Read Res Q. 1988;23(4):454–464. 22. Schraw G, Roedel TD. Test difficulty and judgment bias. Mem Cogn. 1994;22(1):63–69. 23. Nietfeld JL, Schraw G. The effect of knowledge and strategy training on monitoring accuracy. J Educ Res. 2002;95(3):131–142. 24. Dunlosky J, Thiede KW. Four cornerstones of calibration research: why understanding students' judgments can improve their achievement. Learn Instr. 2013;24(1):58–61. 25. Schraw G. A conceptual analysis of five measures of metacognitive monitoring. Metacog Learn. 2009;4(1):33–45. 26. Dunlosky J, Ariel R. Self-regulated learning and the allocation of study time. Psychol Learn Motiv. 2011;54:103–140. 27. Dunlosky J, Rawson KA. Overconfidence produces underachievement: inaccurate self evaluations undermine students' learning and retention. Learn Instr. 2012;22(4):271–280. 28. Persky AM, Greene JM, Anksorus H, Fuller KA, McLaughlin JE. Developing an innovative, comprehensive first-year capstone to assess and inform student learning and curriculum effectiveness. Am J Pharm Educ. 2017 [published online ahead of print 6 November 2017]. https://www.ajpe.org/doi/pdf/10.5688/ajpe6730 Accessed 24 July 2018. 29. Nelson TO. A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychol Bull. 1984;95(1):109–133. 30. Cohen J. A power primer. Psychol Bull. 1992;112(1):155–159. 31. IBM SPSS Statistics for Windows [Computer Program]. Version 24.0. Armonk, NY: IBM Corp.; 2016. 32. Dunning D, Johnson K, Ehrlinger J, Kruger J. Why people fail to recognize their own incompetence. Curr Dir Psychol Sci. 2003;12(3):83–87.
9
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46.
Hacker DJ, Bol L, Horgan DD, Rakow EA. Test prediction and performance in a classroom context. J Educ Psychol. 2000;92(1):160–170. Schwartz BL, Metcalfe J. Tip-of-the-tongue (TOT) states: retrieval, behavior, and experience. Mem Cognit. 2011;39(5):737–749. Schacter DL. The seven sins of memory: insights from psychology and cognitive neuroscience. Am Psychol. 1999;54(3):182–203. Thiede KW, Griffin TD, Wiley J, Anderson MCM. Poor metacomprehension accuracy as a result of inappropriate cue use. Discourse Process. 2010;47(4):331–362. Koriat A. How do we know that we know? The accessibility model of the feeling of knowing. Psychol Rev. 1993;100(4):609–639. Rawson KA, Dunlosky J. Are performance predictions for text based on ease of processing? J Exp Psychol Learn Mem Cogn. 2002;28(1):69–80. Stoeber J, Hotham S. Perfectionism and social desirability: students report increased perfectionism to create a positive impression. Pers Individ Dif. 2013;55(5):626–629. Ryan AM, Pintrich PR, Midgley C. Avoiding seeking help in the classroom: who and why? Educ Psychol Rev. 2001;13(2):93–114. Wernimont PF, Campbell JP. Signs, samples, and criteria. J Appl Psychol. 1968;52(5):372–376. Motowidlo SJ, Hooper AC, Jackson HL. Implicit policies about relations between personality traits and behavioral effectiveness in situational judgment items. J Appl Psychol. 2006;91(4):749–761. Motowidlo SJ, Beier ME. Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. J Appl Psychol. 2010;95(2):321–333. Patterson F, Prescott-Clements L, Zibarras L, Edwards H, Kerrin M, Cousans F. Recruiting for values in healthcare: a preliminary review of the evidence. Adv Health Sci Educ Theory Pract. 2016;21(4):859–881. Rhodes MG, Tauber SK. The influence of delaying judgments of learning on metacognitive accuracy: a meta-analytic review. Psychol Bull. 2011;137(1):131–148. Tauber SK, Dunlosky J, Rawson KA. The influence of retrieval practice versus delayed judgments of learning on memory resolving a memory-metamemory paradox. Exp Psychol. 2015;62(4):254–263.
10
Contents lists available at ScienceDirect
Currents in Pharmacy Teaching and Learning journal homepage: www.elsevier.com/locate/cptl
Research Note
Assessing metacognition in the classroom: Student help-seeking behavior Youn Chu, Shannon Palmer, Adam M. Persky
⁎
Division of Pharmacotherapy and Experimental Therapeutics, Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, 2312 Kerr Hall, CB#7569, Chapel Hill, NC 27599, United States
A R T IC LE I N F O
ABS TRA CT
Keywords: Academic help-seeking behavior Metacognition Capstone Pharmacy students Social desirability
Introduction: The study's purpose was to develop an assessment of students’ metacognitive monitoring of help-seeking behavior. Methods: This study piloted an assessment of help-seeking behavior in first-year student pharmacists to answer two questions: (1) Does help-seeking behavior depend on how familiar students are with the content? and (2) When students ask for help, does their performance and metacognition differ from when they do not seek help? As part of their year-end capstone, students answered drug information questions. The drugs within these questions were chosen based on the level of emphasis during the first-year curriculum (i.e. more familiar or less familiar). For each question, students rated their confidence level for their answer's correctness and marked whether they would ask their preceptor for help. Bias scores were calculated under conditions of familiarity based on level of emphasis (more familiar vs. less familiar) and help-seeking (asked for help and did not ask for help). Results: Students performed better on more familiar material (d = 1.2), with a small difference in confidence on more familiar material (d = 0.2). When students asked for help, they scored lower (d = −2.2) and reported lower confidence (d = −3.7). Students were more likely to ask for help from their preceptors on more familiar content than less familiar (odds ratio = 1.25) and less likely to ask for help when they were overconfident (odds ratio = 0.18). Conclusions: Overall, students were more overconfident for less familiar material and were less likely to ask for help.
Introduction Imagine the following situation: Elisa is on her hospital introductory pharmacy practice experience (IPPE) and is asked to recommend an appropriate dose of a medication by the rounding team. The answer comes to her mind quickly and she states the dose with confidence. However, the dose that she recommends would have significant adverse effects. This medical error is completely preventable. In this case, Elisa associated how quickly an answer came to mind with “correctness” of the answer which may be the result of poor metacognition.1–3 Instead, she should have asked her preceptor or looked up the dosing information in the appropriate resources. In simple terms, Elisa should have been better aware of what she knew (or did not know), how well she knew it, and how that may relate to her confidence in her knowledge. She should have asked for help. Being cognizant of what one does not know is an important skill in a learner's development.4 Student pharmacists and other health profession students must utilize help-seeking skills
⁎
Corresponding author. E-mail address: [email protected] (A.M. Persky).
https://doi.org/10.1016/j.cptl.2018.08.011 Received 20 February 2018; Received in revised form 19 June 2018; Accepted 3 August 2018 1877-1297/ © 2018 Elsevier Inc. All rights reserved.
Please cite this article as: Chu, Y., Currents in Pharmacy Teaching and Learning (2018), https://doi.org/10.1016/j.cptl.2018.08.011
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
during clinical experiences to fill their gaps in knowledge. Because there are few studies investigating how to assess students’ competency in asking for help, the purpose of this study is to pilot a metacognitive assessment aimed at characterizing student pharmacists’ likelihood to seek help in preparation for early immersion experience. Help-seeking behavior is a type of metacognitive monitoring or the process a learner uses to evaluate how well they have learned material or performed a task. In this case, the output for the monitoring is help-seeking, a form of metacognitive control, the use of metacognitive monitoring to guide one's future behavior. Help-seeking behavior can be defined as “an achievement behavior involving the search for and employment of a strategy to obtain success”.5 A theoretical framework for help-seeking has been established and consists of five steps: (1) become aware of need for help, (2) decide to seek help, (3) identify potential helper(s), (4) use strategies to elicit help, and (5) evaluate help-seeking episode.6,7 The critical first step of this model challenges the learner to assess task difficulty, monitor their task progress, and evaluate their own comprehension. That is, learners need to use metacognitive monitoring; they need to be aware and understand their own thought process. If learners cannot accurately monitor what they have learned, they will struggle with decisions on what material needs to be studied, for how long, and when to ask for assistance.8–10 In other words, if learners do not have appropriate metacognitive monitoring skills, they will struggle with metacognitive control and acting in their best self-interests (i.e. self-regulated learning). The ability to accurately judge task progress and comprehension requires metacognitive skills. Students must become efficient in this first step of the help-seeking process to develop into selfregulated learners that act in their own, long-term best interest.11 As such, we focused efforts to assess the initial two steps in the help-seeking process, becoming aware for the need of help and deciding to seek help. Studies have found a relationship between help-seeking behavior and academic success. Ryan et al.12 found that with learners as young as sixth grade, adaptive help-seeking behavior has been associated with better outcomes in school. In student pharmacists, academic help-seeking behavior was associated with greater academic competence and more positive faculty/student relationships.13 Due to the positive impact help-seeking behavior appears to have in the classroom, we wanted to explore pharmacy students’ metacognition in relation to help-seeking behavior. That is, are student pharmacists aware enough of their knowledge and skills to know when to ask for help? However, studying help-seeking behavior is complex, impacted by instructional methods, environment, and learners’ self-concepts.14 To date, there has been very little work on the metacognitive monitoring involved in help-seeking. In this current study, we used a written assessment to help evaluate student's metacognitive monitoring of help-seeking behavior. Data on assessing help-seeking behavior through assessment is very limited, if not completely absent. However, there is a plethora of literature on metacognitive monitoring using judgments of learning.15–17 A judgment of learning (JOL) is defined as one's perceptions of one's own ability. This current study and its methods are an extension of this literature but applied to a follow-up action, helpseeking. Within the JOL literature, comprehension monitoring is linked to the formation of a JOL and, consequentially, control of selfregulated learning.17,18 As an example, a JOL can evaluate how well one has understood information or performed a task; this can include scoring one's own confidence after being asked a question.17,18 In addition, tutoring systems have been used to look at using feedback to improve students’ help-seeking skills, particularly in younger students.7 These systems provide hints as students proceed through the program. These hints help develop metacognitive behaviors and improved help-seeking skills by giving the learner cues to change their behavior.7 For example, one system, aimed at high school geometry instruction, used a cognitive model to track students in their various approaches through geometry problems, estimating their knowledge level, and providing domain-level feedback and on-demand hints regarding the steps of these problems. This tutoring system indicated that help-seeking behavior can be trained within a student population even if it is not obviously stated. Although we are not assessing transferability of the implicit learning in this study, we hope that the same implicit effects seen within the tutoring system literature can be seen with this written assessment as it might impact performance in the experiential part of the curriculum. During IPPEs and advanced pharmacy practice experiences (APPEs), student pharmacists are given tasks and problems they have not covered in school, yet they are expected to use available resources, including their preceptors, to arrive at the correct answer. We expect students will ask for help when they have less background knowledge (less familiar material) than material they have prior exposure to within the curriculum. Thus, it is important to assess students’ help-seeking behavior under conditions of more familiar and less familiar information. Stavrianopoulos found students with better metacognitive monitoring abilities reviewed material they had previously indicated as unfamiliar.19 This supports the work about error reduction proposed by Metcalfe et al.20 that students will study material with the biggest difference between their JOL (how well they think they know a topic) and their actual performance. By looking at material that students had varying levels of familiarity with (i.e. familiar vs. less familiar material), we examined the impact on students’ metacognition and help-seeking behavior. Besides the issue of familiarity, we also expect that the learners with higher degrees of overconfidence (predicted score greater than actual score) are less likely to ask for help compared to students with stronger degrees of underconfidence (predicted score less than actual score). In this study, we measured calibration or metacognitive accuracy through the difference in confidence judgments and performance. The accuracy of metacognitive monitoring can be impacted by various factors including problem difficulty, background (or prior) knowledge (i.e. more familiar or less familiar information) and desired performance level.21–23 Bias is the difference between the confidence judgment and actual performance and includes both directionality (i.e. under- or overconfident) and magnitude of that directionality; absolute bias captures the magnitude or accuracy only.24,25 In prior research, overconfidence has been associated with early termination of study and poorer retention of material.18,19 Therefore, we may expect that students with more overconfidence would be less likely to ask for help.26,27 Taken together, this study piloted an assessment of academic help-seeking behavior of student pharmacists at the end of their first year and getting ready to transition to their first experiential setting. There were two primary questions: (1) Does help-seeking behavior depend on content familiarity and (2) When students ask for help, do their performance and confidence differ from when they do not ask for help? Several subquestions will be addressed that examine metacognitive monitoring for higher or lower 2
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
performing students and to identify factors that may predict help-seeking behavior. The overall goal is to develop a process to assess students metacognitive monitoring of help-seeking. Methods The help-seeking behavior assessment was within the find-and-apply assessment of the first-year capstone at the University of North Carolina Eshelman School of Pharmacy. The goal of the capstone assessment was to provide feedback to students, preceptors, and the school. As part of a new curriculum, we developed a capstone assessment administered after the first year to help prepare students for their first early immersion experience (e.g., IPPE). The capstone is described elsewhere and included four components: (1) a closed book knowledge assessment, (2) an open resource find-and-apply assessment, (3) an assessment of interpersonal skills, and (4) a debriefing session.28 The find-and-apply assessment is an open-resource assessment of students’ ability to answer drug information questions by finding resources and applying their knowledge and interpretation of those resources. Students were given four hours to complete this assessment. Questions were multiple-choice with a single best correct answer and ranged from identifying key adverse effects and counseling a patient to drug substitutions and identifying the severity of drug interactions. The questions were developed to stretch the students’ ability by not focusing on recallable facts but focusing on utilizing drug information resources (e.g., PubMed, Micromedex, UpToDate) to answer unfamiliar questions. Students could use any resource on their computer or internet including electronic library resources. This section contained 73 questions with 48 questions related to the current study. Of these 48 questions, 16 were drug information questions and 32 questions were used to assess help-seeking behavior. For the 16 drug information questions, assessed drugs were selected based on the school's curriculum map. Eight drugs were selected because they were discussed in multiple courses with some depth including being assessed. This was deemed the more familiar material. Another eight drugs were selected because they were either used as examples only or were not discussed within first-year courses. These were deemed the less familiar drugs. While these less familiar drugs may be more variable in familiarity because of students work or clinical experience outside of class, we were certain based on course work they were not emphasized during formal instruction and should still be unfamiliar enough for purposes of this study. The 16 drug information questions were graded as correct or incorrect with no partial credit. For the 32 questions used to assess help-seeking behavior, after answering the drug information question, students were asked to make a confidence judgment (continuous scale of 0–4 where 0 = no confidence and 4 = high confidence or 100% confident) on the correctness of their answer. They were then asked if they would ask their preceptor for help on that question (yes or no). Appendix 1 has an example of this question format. This study assessed students help-seeking relative to their judgment accuracy (how well a person’ s judgments are related to target performance). The absolute accuracy is the magnitude of the effect, because it reflects the absolute match between judgment magnitude and target performance.24 Relative accuracy is the degree to which the judgments discriminate between different levels of performance across items.24 These two measures are independent as a student can have good absolute accuracy but poor relative accuracy, and vice versa.24,25 While there are various methods to calculate absolute and relative accuracy, we used bias, absolute bias, and the correlation coefficient, gamma.24 For each group of questions (more familiar vs. less familiar; ask preceptor or not ask preceptor), scores were averaged as percent correct (e.g., 60% correct). Confidence judgments were also averaged and converted to a percentage (e.g., 70% confident). Bias score was calculated by the difference in confidence (as a percent) and correctness (as a percent) (bias = confidence − score). This is a measure of overconfidence (confidence greater than actual score) or underconfidence (confidence less than actual score). Negative values of bias are considered underconfident and positive values are considered overconfident. For this study we also defined a third category called “well-calibrated” for confidence judgment within 20% of performance (e.g., the average score was 70% but average confidence was 60–80%). In addition, an absolute bias score was calculated (absolute bias = absolute value of bias) which indicates the magnitude of the difference between the confidence judgment and performance regardless of directionality. Relative accuracy for each individual was computed as a gamma correlation between the confidence judgments and actual performance.29 This was a within-subjects design because students answered both familiar and less familiar questions and asked or did not ask for help. As such, a paired t-test was used to determine differences between familiarity conditions (more/less) and asking for preceptor help (yes/no). Logistic regression was used to determine potential predictors of help-seeking behavior. This was performed with the dependent variable being asking for help (yes/no) and the independent variables being the average question score (continuous variable), average question confidence (continuous variable), familiarity of the topic (more/less), and bias (well-calibrated, underconfident, or overconfident). All variables were entered together. An odds ratio (OR) was calculated when appropriate using chisquare and a Cohen's d was used to determine effect sizes. For Cohen's d, a d < 0.4 was considered small, 0.4 < d < 0.7 was considered medium, and d > 0.7 was considered large.30 Finally, students were placed into tertiles based on examination performance and comparisons were made on performance and metacognitive metrics using an ANOVA with Tukey post-hoc test. Statistical analyses were performed utilizing SPSS31 and significance was set at p < 0.05. The study was deemed exempt by the university's institutional review board. Results Of the 143 first-year student pharmacists who participated in the study, 100 students had complete data sets; the excluded students did not answer one or more questions of confidence or asking a preceptor. The first question we sought to answer was “Does help-seeking behavior depend on content familiarity?”. To answer this question, 3
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
Fig. 1. Performance and confidence based on familiarity. Data presented as mean percent and standard deviation. Students’ scores were higher for more familiar content but a small difference in confidence.
we first explored differences in performance and confidence based on content familiarity. For more familiar material compared to less familiar material, students performed better (d = 1.2) and had higher confidence (d = 0.22), although not the same degree (Fig. 1). Overall, for familiar content, students had better relative accuracy (gamma = 0.62 vs 0.37) than less familiar content. Students were underconfident on the more familiar material than on the less familiar material (d = 0.87, p < 0.001). There was no difference in accuracy of judgment (i.e., absolute bias, extent of difference between confidence and score) between the two conditions. Finally, when faced with more familiar content, students were more likely to ask a preceptor for help [OR 1.25 (1.01, 1.56), p < 0.05]. The second question asked “When students ask for help, do their performance and confidence differ from when they do not ask for help?”. Only five students would not ask their preceptor for help on any question. On average, students would ask a preceptor for help on 71% of the questions (11 ± 3 questions, range 1–16). For questions which students would ask a preceptor for help, students scored lower (d = −2.2) and were less confident (d = −3.7) (Fig. 2). Overall, students’ relative accuracy was better when they asked the preceptor for help (gamma = 0.54 vs. 0.28). Students who asked for help on the question were more underconfident (d = 0.61, p < 0.001) and less accurate (d = 0.43, p = 0.005). The first two questions raised a third: “What factors may predict the desire to ask for help?” We performed an exploratory logistic regression with the dependent variable of asking a preceptor for help (yes or no). The question score, confidence level, topic familiarity, and confidence category (calibrated, underconfident or overconfident) were entered into the model. The model explained 85% of the variability (Nagelkerke R2) (Table 1). Students were less likely to ask for help with less familiar content than more familiar content. As performance and confidence increased, the odds of students asking for help decreased. For a 1% increase in score (e.g., 50% to 51%), the odds a student would ask for help decreased by 4% when other factors are held constant. For a 1% increase in confidence (70% to 71%), the odds of asking for help decreased by 14% when other factors are held constant. Finally, compared to well-calibrated students (confidence within ± 10% points of performance), overconfident students were less likely to ask for help but underconfident students were equally likely to ask for help (Table 1, Fig. 3). This suggests more familiar material influenced students’
Fig. 2. Performance and confidence for whether students asked a preceptor for help. Data presented as mean percent and standard deviation. Students’ scores and confidence were higher when asking for help than when not asking for help. 4
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
Table 1 Summary of logistic regression. Metric
Standard
Score (%) Confidence (%) Less familiar Overconfident Underconfident
– – More familiar Well-Calibrated
−.046 −.15 −1.6 −1.7 −1.2
Odds ratio
95% Confidence interval
0.96a 0.86a 0.20a 0.18a 0.30
0.92, 0.99 0.82, 0.90 0.067, 0.59 0.036, 0.93 0.047, 1.2
a
p < 0.05. Scores = performance on the questions; Confidence = confidence students had on their answers; Less familiar = content that was less familiar to students because it was less emphasized in the curriculum; Overconfident = determined by confidence greater than score; Underconfidence = confidence less than score; Well-calibrated = when the confidence and scores were closely matched.
Fig. 3. Confidence versus performance and zones of help seeking. Zone I = Underconfidence; Zone 2 = Well-calibrated; Zone III = Overconfidence. Open-circles = average score and confidence on questions students asked for help; Closed circles = average score and confidence for questions students did not ask for help on.
likelihood to ask for help more than their level of confidence or how well they performed. We further explored data based on student performance because students who perform lower also tend to be less metacognitively aware.32,33 As such, our final set of questions probed if better performing students behave differently than lower performing students. Students were separated into tertiles based on performance on the 16 content-related capstone questions. When examining the impact of familiarity on help-seeking, the lowest tertile had lower confidence compared to the top tertile. The lowest performing students also had higher bias scores (i.e. more positive/less negative) and lower absolute bias compared to the top two tertiles (Fig. 4A, Table 2). For the less familiar material, only the lowest tertile had higher bias compared to higher and middle performing students (Fig. 4B, Table 2). There were no differences with respect to confidence ratings between the tertiles. We were also interested in learning if lower performing students had differential metacognitive monitoring when asking for help. The lowest performing students had lower confidence, higher bias scores, and lower absolute bias scores when they asked for help (Fig. 4C, Table 2). There were no differences in metacognitive monitoring for questions on which students did not ask for help (Fig. 4D, Table 2).
Discussion The purpose of this study was to examine a novel way to assess the metacognitive monitoring of help-seeking behavior. As expected, students performed better on more familiar material than less familiar material and reported slightly higher confidence on more familiar material. Surprisingly, students were less likely to ask for help on less familiar than more familiar material which may support the “uninformed and unaware” phenomena.34 Students were less likely to ask a preceptor for help if they were overconfident, especially for unfamiliar material. This may reflect the Dunning-Kruger effect which is a cognitive bias wherein individuals hold an overly optimistic and miscalibrated view about their abilities. Low knowledge individuals overestimate their ability (i.e. overconfident) and high knowledge individuals will underestimate their abilities (i.e. underconfident). We found that students were more likely to be underconfident on more familiar material (i.e. skilled and aware). The more familiar material is subject to memory error, which may reduce confidence level.35 That is, students were aware they once knew the material but are unaware of their current knowledge. Since the familiar drugs were taught and 5
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
Fig. 4. Performance and confidence based on tertiles of performance. (A) More familiar material; (B) Less familiar material; (C) When students would ask a preceptor for help; (D) When students would not ask a preceptor for help. The bottom tertile generally had differences in bias and absolute bias compared to the upper two tertiles. Table 2 Summary of parameters when students were asked about more familiar or less familiar and asked for help content based on performance on the questions. Condition
Tertile
Score
Confidence
Bias
Absolute Bias
More familiar
Top (n = 27) Middle (n = 30) Bottom (n = 43) Top (n = 27) Middle (n = 29) Bottom (n = 40) Top (n = 27) Middle (n = 30) Bottom (n = 43) Top (n = 27) Middle (n = 29) Bottom (n = 40)
84 77 65 72 60 44 70 63 44 96 88 86
64 58 57 60 56 54 50 46 43 91 89 87
−20 (13) −20 (11) −8 (15)a,b −12 (17) −4 (18) 10 (15)a,b −20 (17) −17 (14) −0.8 (13)a,b −5.2 (12) 0.3 (17) 1.2 (17)
21 (12) 20 (10) 13 (12)a,b 16 (13) 15 (12) 14 (12) 22 (14) 18 (12) 10 (8.3)a,b 9.3 (8.8) 12 (12) 12 (12)
Less familiar
Asked Preceptor
Did not ask preceptor
(9.0) (7.7) (12)a,b (8.1) (7.7)a (13)a,b (9.0) (11)a (9.7)a,b (7.9) (16) (17)a
a
(12) (12) (11)a (13) (13) (14) (13) (11) (11) (8.9) (11) (14)
p < 0.05 vs. Top Tertile; bp < 0.05 vs. Middle Tertile. Tertile: based on student performance; Bias = difference in confidence and score; Absolute bias = absolute value of bias. 6
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
applied in various courses at different times, they are more likely to be influenced by both proactive and retroactive interference. Conversely, for less familiar drugs, students had little experience with the drugs and assumed they could accurately find and utilize information (i.e. unskilled and unaware). Less familiar material has less chance for memory error because students had very limited exposure to these drugs, therefore, they reported more accurate confidence scores matching the performance score. Ultimately the overconfidence can be explained with how students use cues to predict how well they understand the material. In our early example, Elisa used a cue of how fast the answer came to mind as a cue to correctness of her answer. Overconfident JOLs (i.e. confidence judgments) are often explained by the cue utilization approach which states that for a learner to monitor their comprehension, they utilize cues that they view as predictive on how well they comprehend. That is, the learner has some internal check and balance system. However, research on cue utilization indicates that learners often have low comprehension monitoring skills because they frequently utilize cues that do not possess a high degree of predictive validity.36 In other words, their internal cues do not predict their actual comprehension. Therefore, it is possible, the ease at which learners found or recalled information suggests to them their answer is correct when in fact, it may not be. This ease of fluency has been associated with higher confidence judgments but lower or no differences in performance.21,37,38 In contrast, deeply comprehended information assumes the information has been sufficiently integrated into the learners’ prior knowledge and it is likely that learners are able to recall it. What this means is, we assume that the amount of recallable information often exceeds the amount of well comprehended information and, when using memory-based cues for comprehension monitoring, it can lead to overconfidence. When seeking help, this study found that familiarity was a strong predictor, more so than performance, confidence, or calibration. Students were more likely to ask for help on more familiar material than less familiar material, which contradicts our initial expectation but in hindsight may make sense. On one hand, if students had appropriate metacognitive accuracy, they would know when they encountered a new situation and ask for help. However, because students had knowledge of the topics, they were more aware of their limits and would ask for help. For less familiar material, students had little background knowledge which may lead to being unaware of the need for help. In addition, students may avoid asking their preceptor questions for which they have no or little baseline information, a form of social desirability.39 In this case, the hope is students would look up the drug information prior to approaching their preceptors for confirmation or clarification. For example, learners may not ask for help out of fear that they will receive less credit for a successful outcome or that the teacher or their peers will view them as incompetent.6,30 This may explain why students asked for help on over 70% of the questions, they may feel that is the socially acceptable response and do not want to appear incompetent.6,40 To put this study back into instructional context, this part of the capstone was designed to help provide feedback to students moving forward into their first clinical experiences. During the debrief session of the capstone, the preliminary data for the study was shown and discussed, especially about the less familiar content and being overconfident. The goal was to help bring awareness of student biases and discuss when it is appropriate to ask for assistance. A secondary goal was to reinforce the help-seeking behavior. As noted in the help-seeking literature, there is an implicit learning that occurs with asking for help.7 The question arises, “Does asking for help on a paper and pen assessment translate to actual help-seeking behavior?”. The answer is potentially with support from two frameworks. The behavioral consistency theory states that past behavior is the best predictor of future behavior. If asking for help in this assessment is an example of current behavior, then that allows prediction of future behavior within experiential training.41,42 The second framework is the implicit trait policies (ITPs) which proposes that individuals develop beliefs about the effectiveness of different behavior. That is, in each situation, individuals weigh the costs and benefits associated with expressing certain traits (i.e., asking for help) in context to an individuals’ inherent tendencies or traits, and dependent on the job level, specific job knowledge.43,44 For example, a pharmacist dealing with a sensitive situation may have to make a judgment about the utility (i.e. costs/benefits) of demonstrating empathy and agreeableness as a more successful strategy than acting brusquely or being disagreeable even if the pharmacist's preference tends towards being generally disagreeable and/or empathetic. There are several limitations to the current study. First, familiarity was based on the extent the drugs were discussed within the curriculum based on the school's curriculum map. It is difficult to quantify what justifies familiarity and it does not include exposure students may have outside of school through extra- and co-curriculars. However, based on performance, familiar material scored higher thus potentially confirming its familiarity to students. This also was confirmed by asking students to provide a rating of familiarity. Due to metacognitive bias. that students view of familiarity may not be the same as the expectations of the instructors’, we did not use the student ranking of familiarity. This ranking was there for confirmation purposes only, not to classify content. The second area is that help-seeking questions could be biased because of social desirability and it may be helpful in the future to have some measure of social desirability as an explanatory factor or maybe limit the opportunities for help-seeking to help assign more value to the strategy (i.e. make it a limited resource). The third limitation is students indicated they would ask a preceptor for help but there was no measure of them seeking help. However, there are at least two theories stated earlier that suggest students may be likely to ask for help based on the paper-and-pen assessment There are a couple of instructional implications based on the results of this study. The first implication is preparing health profession students to become clinicians involves cultivating both knowledge and behavioral skills. It is crucial for instructors and administrators to develop tools to measure student progress and ability across categories. This study provides an example of how assessments may be used to help measure students’ metacognition. These types of assessments provide a potential way to measure student growth over time. Schools of pharmacy could use help-seeking assessments pre- and post- immersion experiences to help measure if improvement occurred in this skill over the course of the experience. This could help demonstrate if students’ judgment of when to ask for help improved after seeing clinical decisions made in real-word settings. The second implication could be helping learners utilize confidence ratings on their own so they can avoid the ramifications of overconfidence and being “unskilled and unaware.” These confidence judgments tend to be more impactful if they are delayed versus made immediately.45,46 Regardless, these 7
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
judgments may include learners keeping a record of topics, rating their confidence on those topics, and, for new topics, checking their knowledge before trying to apply that knowledge. This would provide another method for students to visualize if they were overconfident or underconfident with responses. It should also be noted that even though students may have high levels of confidence, it does not mean they are going to make appropriate decisions regarding help-seeking. In this study, students had high levels of confidence but when comparing their confidence to their performance, their performance was greater than their confidence (i.e., overconfident). It was these students that would be more likely to seek help. Conclusions This study was successful in describing the metacognitive monitoring of help-seeking behavior. The study captures theoretically what students would do but it may be confounded by some aspects of social desirability. Regardless, follow-up studies may focus on the differences investigating further when students seek help and potential differences between seeking help from a preceptor versus a teacher, peer, or mentor. Conflict of interest None. Disclosures None. Acknowledgments The authors would like to thank the team that helped develop the capstone, especially: Jessica Greene and Kathryn Fuller who were instrumental in the Find-and-Apply assessment development. The authors also thank Sabrina Dunham for her review of the manuscript. Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.cptl.2018.08.011. Appendix 1. Sample drug information question KF is a 45-year old female with heartburn and indigestion. She was just started on prednisone 7.5 mg PO daily by her rheumatologist to help control rheumatoid arthritis (RA) disease activity. At her rheumatology appointment, her RAPID-3 score was 6. Past medical history of RA and type-1 diabetes. Meds
• Levothyroxine 100 mcg daily • Methotrexate 15 mg once weekly with folic acid 3 mg once daily • Acetaminophen 650 mg every 8 hours PRN • Celecoxib 100 mg twice daily • Calcium citrate 1500 mg/day • Multivitamin once daily • Insulin glargine 22 units once daily • Insulin lispro 6 units with breakfast, 7 units with lunch and 9 units with dinner • Prednisone 7.5 mg once daily X 1 week then 5 mg once daily X 1 week Allergies and ADRs: sulfonamide antibiotics (Rash); azithromycin (abdominal pain); amoxicillin (diarrhea) Blood Glucose log
Monday Tuesday Wednesday Thursday Friday
Before Breakfast
Before Lunch
128 103 87 103 74
127 78 72 90 125
Low BG symptoms
55 (3:46 p.m.) 60 (4:52 p.m.) 58 (4:07 p.m.) 8
Before Dinner
Bedtime
72 195 88 192 201
121 102 105 101 97
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
Saturday Sunday
88 83
98 88
62 (4:29 p.m.)
93 188
96 122
You are asked to counsel KF on the use of her prednisone. What would be the top priorities to counsel her on? A Water retention and development of a “moon face” B Elevated morning glucose and vivid dreams C Risk of osteoporosis and flu-like symptoms For the question you just answered, what is your level of confidence that your answer is correct? A B C D E
0 – no confidence 1 2 3 4 – high confidence For the question above, would you ask your preceptor for assistance?
A Yes B No PO = by mouth; PRN = as needed.
References 1. Benjamin AS, Bjork RA, Schwartz BL. The mismeasure of memory: when retrieval fluency is misleading as a metamnemonic index. J Exp Psychol Gen. 1998;127(1):55–68. 2. Serra MJ, Dunlosky J. Does retrieval fluency contribute to the underconfidence-with-practice effect? J Exp Psychol Learn Mem Cogn. 2005;31(6):1258–1266. 3. Koriat A, Ma'ayan H. The effects of encoding fluency and retrieval fluency on judgments of learning. J Mem Lang. 2005;52(4):478–492. 4. McLaughlin JE, Bush AA, Rodgers PT, et al. Exploring the requisite skills and competencies of pharmacists needed for success in an evolving health care environment. Am J Pharm Educ. 2017;81(6) https://doi.org/10.5688/ajpe816116. 5. Ames R, Lau S. An attributional analysis of student help-seeking in academic settings. J Educ Psychol. 1982;74(3):414–423. 6. Gall SN. Help-seeking: an understudied problem-solving skill in children. Dev Rev. 1981;1(3):224–246. 7. Roll I, Aleven V, McLaren BM, Koedinger KR. Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learn Instr. 2011;21(2):267–280. 8. de Bruin ABH, Dunlosky J, Cavalcanti RB. Monitoring and regulation of learning in medical education: the need for predictive cues. Med Educ. 2017;51(6):575–584. 9. Soderstrom NC, Clark CT, Halamish V, Bjork EL. Judgments of learning as memory modifiers. J Exp Psychol Learn Mem Cogn. 2015;41(2):553–558. 10. Townsend CL, Heit E. Judgments of learning and improvement. Mem Cogn. 2011;39(2):204–216. 11. Puustinen M. Help-seeking behavior in a problem-solving situation: development of self-regulation. Eur J Psychol Educ. 1998;13(2):271–282. 12. Ryan AM, Shin H. Help-seeking tendencies during early adolescence: an examination of motivational correlates and consequences for achievement. Learn Instr. 2011;21(2):247–256. 13. Payakachat N, Gubbins PO, Ragland D, et al. Academic help-seeking behavior among student pharmacists. Am J Pharm Educ. 2013;77(1) https://doi.org/10.5688/ ajpe7717. 14. Strategic Help Seeking: Implications for Learning and Teaching. St. Louis, MO: Routledge; 2012. 15. Handbook of Metamemory and Memory. New York, NY: Psychology Press, Taylor & Francis Group, LLC; 2008. 16. Dunlosky J, Metcalfe J. Metacognition. Los Angeles, CA: SAGE Publications, Inc.; 2008. 17. Metcalfe J. Metacognitive judgments and control of study. Curr Dir Psychol Sci. 2009;18(3):159–163. 18. Koriat A, Goldsmith M. Monitoring and control processes in the strategic regulation of memory accuracy. Psychol Rev. 1996;103(3):490–517. 19. Stavrianopoulos K. Adolescents' metacognitive knowledge monitoring and academic help seeking: the role of motivation orientation. Coll Stud J. 2007;41(2):444–453. 20. Metcalfe J, Kornell N. A region of proximal learning model of study time allocation. J Mem Lang. 2005;52(4):463–477. 21. Pressley M, Ghatala ES. Delusions about performance on multiple-choice comprehension tests. Read Res Q. 1988;23(4):454–464. 22. Schraw G, Roedel TD. Test difficulty and judgment bias. Mem Cogn. 1994;22(1):63–69. 23. Nietfeld JL, Schraw G. The effect of knowledge and strategy training on monitoring accuracy. J Educ Res. 2002;95(3):131–142. 24. Dunlosky J, Thiede KW. Four cornerstones of calibration research: why understanding students' judgments can improve their achievement. Learn Instr. 2013;24(1):58–61. 25. Schraw G. A conceptual analysis of five measures of metacognitive monitoring. Metacog Learn. 2009;4(1):33–45. 26. Dunlosky J, Ariel R. Self-regulated learning and the allocation of study time. Psychol Learn Motiv. 2011;54:103–140. 27. Dunlosky J, Rawson KA. Overconfidence produces underachievement: inaccurate self evaluations undermine students' learning and retention. Learn Instr. 2012;22(4):271–280. 28. Persky AM, Greene JM, Anksorus H, Fuller KA, McLaughlin JE. Developing an innovative, comprehensive first-year capstone to assess and inform student learning and curriculum effectiveness. Am J Pharm Educ. 2017 [published online ahead of print 6 November 2017]. https://www.ajpe.org/doi/pdf/10.5688/ajpe6730 Accessed 24 July 2018. 29. Nelson TO. A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychol Bull. 1984;95(1):109–133. 30. Cohen J. A power primer. Psychol Bull. 1992;112(1):155–159. 31. IBM SPSS Statistics for Windows [Computer Program]. Version 24.0. Armonk, NY: IBM Corp.; 2016. 32. Dunning D, Johnson K, Ehrlinger J, Kruger J. Why people fail to recognize their own incompetence. Curr Dir Psychol Sci. 2003;12(3):83–87.
9
Currents in Pharmacy Teaching and Learning xxx (xxxx) xxx–xxx
Y. Chu et al.
33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45. 46.
Hacker DJ, Bol L, Horgan DD, Rakow EA. Test prediction and performance in a classroom context. J Educ Psychol. 2000;92(1):160–170. Schwartz BL, Metcalfe J. Tip-of-the-tongue (TOT) states: retrieval, behavior, and experience. Mem Cognit. 2011;39(5):737–749. Schacter DL. The seven sins of memory: insights from psychology and cognitive neuroscience. Am Psychol. 1999;54(3):182–203. Thiede KW, Griffin TD, Wiley J, Anderson MCM. Poor metacomprehension accuracy as a result of inappropriate cue use. Discourse Process. 2010;47(4):331–362. Koriat A. How do we know that we know? The accessibility model of the feeling of knowing. Psychol Rev. 1993;100(4):609–639. Rawson KA, Dunlosky J. Are performance predictions for text based on ease of processing? J Exp Psychol Learn Mem Cogn. 2002;28(1):69–80. Stoeber J, Hotham S. Perfectionism and social desirability: students report increased perfectionism to create a positive impression. Pers Individ Dif. 2013;55(5):626–629. Ryan AM, Pintrich PR, Midgley C. Avoiding seeking help in the classroom: who and why? Educ Psychol Rev. 2001;13(2):93–114. Wernimont PF, Campbell JP. Signs, samples, and criteria. J Appl Psychol. 1968;52(5):372–376. Motowidlo SJ, Hooper AC, Jackson HL. Implicit policies about relations between personality traits and behavioral effectiveness in situational judgment items. J Appl Psychol. 2006;91(4):749–761. Motowidlo SJ, Beier ME. Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. J Appl Psychol. 2010;95(2):321–333. Patterson F, Prescott-Clements L, Zibarras L, Edwards H, Kerrin M, Cousans F. Recruiting for values in healthcare: a preliminary review of the evidence. Adv Health Sci Educ Theory Pract. 2016;21(4):859–881. Rhodes MG, Tauber SK. The influence of delaying judgments of learning on metacognitive accuracy: a meta-analytic review. Psychol Bull. 2011;137(1):131–148. Tauber SK, Dunlosky J, Rawson KA. The influence of retrieval practice versus delayed judgments of learning on memory resolving a memory-metamemory paradox. Exp Psychol. 2015;62(4):254–263.
10