- Email: [email protected]
Implementing Survey-Based Changes and Analyzing Usage in an Online Curriculum for Surgical Interns
j o u r n a l o f s u r g i c a l r e s e a r c h o c t o b e r 2 0 1 9 ( 2 4 2 ) 8 7 e9 3
Available online at www.sciencedirect.com
ScienceDirect journal homepage: www.JournalofSurgicalResearch.com
Implementing Survey-Based Changes and Analyzing Usage in an Online Curriculum for Surgical Interns Mohammadali E. Alishahedani, BA, George A. Sarosi, MD, and Janice A. Taylor, MD, MEd* Department of Surgery, University of Florida College of Medicine, Gainesville Florida
article info
abstract
Article history:
Background: To improve the online curriculum at the authors’ program, study habits and
Received 13 November 2018
learning resources of surgical interns were evaluated. Based on the results, changes were
Received in revised form
implemented to align with their teaching conferences. We sought to determine utilization
2 April 2019
and satisfaction with the new materials.
Accepted 9 April 2019
Methods: At the end of the 2017 academic year, surgical interns at a single institution
Available online xxx
voluntarily responded to an Institutional Review Boardeapproved survey regarding new and established learning material use and study habits. Responses were deidentified.
Keywords:
Descriptive statistics were performed on demographics. Likert responses underwent Mann
Surgical education
eWhitney analysis (a ¼ 0.05).
Online learning
Results: The response rate was 52.9% (n ¼ 9). The internet was the most-used resource
Study habits
(P < 0.05). All respondents used the internet to varying degrees. Textbooks and the internet were always used by 22.2%. 33.3% never used textbooks to study. There was a statistically significant increase in internet access but not with other materials (P < 0.05). Regarding new material organized from the pilot, none was consistently accessed. 55.6% sometimes used new weekly reading links associated with a question bank. 66.7% were somewhat satisfied with these links. 44.4% were somewhat satisfied with new video links, organized in parallel with This Week in Surgical Council on Resident Education. Limited sample size did not allow for meaningful statistical analysis of material use with American Board of Surgery In-Training Examination scores. Conclusions: Overall, materials organized based on pilot study feedback were not widely used, and satisfaction was limited. The statistically significant increase in use of internet resources warrants attention. Focusing changes in future curricular design may help create a more effective learning environment. ª 2019 Elsevier Inc. All rights reserved.
Introduction The role of the internet is increasing in surgical education.1-4 Traditional resources, such as “classic” textbooks like Schwartz’s Principles of Surgery and Greenfield’s Surgery,
while still widely used and referenced to build one’s knowledge base, are joined by their electronic counterparts. Internet-based learning platformsdvideo sites, interactive question banks, and customizable reading schedulesdadd to the figurative library of the surgical trainee. Previously, Purcell
* Corresponding author. Department of Surgery, University of Florida College of Medicine, 1600 SW Archer Rd, PO Box 100119, Gainesville FL 32610. Tel.: þ15135828316; fax: þ1 352 273 8772. E-mail address: [email protected] (J.A. Taylor). 0022-4804/$ e see front matter ª 2019 Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jss.2019.04.044
88
j o u r n a l o f s u r g i c a l r e s e a r c h o c t o b e r 2 0 1 9 ( 2 4 2 ) 8 7 e9 3
outlined the evolution of technology in learning resources.5 As surgical trainees use materials outside of their institutional curriculum, there is not only a need to understand their access and study habits but also a need to offer appropriate resources that will be used regularly.4,6,7 One widely known incorporation of internet-based learning into an organized, standardized curriculum is Surgical Council on Resident Education (SCORE), the program put forth by the SCORE.1 Since its inception, SCORE has integrated various learning tools to foster a better learning environment for the surgical trainee. In 2015, SCORE introduced This Week in Score, an organized 2-y curriculum for surgical residents, which acts as a guide through the site’s material for the user.4 Although SCORE is the recognized standard among online surgical curricula, and is the backbone of the authors’ residency teaching schedule, it is used along with other material based on the residents’ study preferences, access capabilities, and individual institutional account. To this end, at the authors’ institution, there are multiple online resources provided to the surgical interns, which include e-book and peerreviewed journal access provided by the university. In an effort to improve the didactic experience of the surgical interns at the authors’ institution, intern classes were surveyed at the end of the 2015 and 2017 academic years. Resource utilization and satisfaction were analyzed in the 2015 pilot study. Based on the results, changes were made to the interns’ organized teaching curriculum. The intent was to introduce new online study tools that would correlate with what interns stated they would use, or desired access to, to give access to different media- or problem-based content to promote more efficient learning, more consistent reading, and increase access to quality just-in-time learning. Then, a follow-up survey was performed in 2017. We expected an increase in intern satisfaction and resource utilization. In addition, we hypothesized that this would have a positive effect on objective markers such as the annual in-service examination.
Material and methods Educational materials Based on 2015 pilot study feedback regarding study material use, 1-y subscriptions were purchased to Scientific American Surgery (SAS; https://www.deckerip.com/products/scientificamerican-surgery/) and COACH Surgery (COACH; https:// collectedmed.com/coachsurgery). SAS provided an extensive question bank and problem-based learning formats; COACH provided multimedia content such as curated video and animation links. The decision to subscribe to these sites was made with input from the Executive Education Committee of the University of Florida, Department of Surgery. For 1 y, interns were sent weekly links from SAS, with questions, based on their department educational conference topics. On average, links from COACH were emailed every other week, based on the This Week in SCORE curriculum used by the interns. The interns had general access to COACH and SAS for self-studying purposes. The departmental educational website also had high-yield links to SCORE (https://www.
surgicalcore.org/), organized based on the weekly conference schedule. Where available, website access histories were used to retrieve data regarding intern activity. For example, both COACH and SAS tracked the number of logins and dates, and which area of the sites were accessed, but COACH also tracked the time of day and length of time for each login. SCORE logged date, time, and visit duration of each access to the site, but access to the high-yield links from SCORE that were added the department educational website were not trackable.
Survey tool and recruitment To evaluate utilization and satisfaction of the new curriculum, a survey tool was used, exempted by the University of Florida Institutional Review Board. The 20-question survey was administered to surgical interns at the University of Florida in Gainesville, at the end of the 2016-2017 academic year (Appendix). The survey was verified by a modified Delphi technique and was based on previously published surveys.6,8 Recruitment for the survey was based on participation during one of the residents’ weekly mandatory general meetings in June. The study team presented the survey tool and study purpose. Participation was voluntary. The survey was administered in pencil/paper fashion. Those who participated were not required to write their name on the survey. There were a total of 17 interns in the program. The study team left the meeting room during completion of the survey by the interns, to minimize any perceived outside influence on responses. It was collected by the designated study gatekeeper, who also deidentified the results.
Statistical analysis In responding to the survey, the interns were asked to consider different aspects of their study habits and resource use. Responses were recorded based on multiple choice, Likert scale, and free text. When interns completed the name field, the study gatekeeper was able to correlate their deidentification code with in-service performance and access history to the online resources, COACH and SAS. Descriptive statistics were performed on the demographic data. KruskaleWallis and ManneWhitney analyses were performed on the Likert responses. Statistics were performed using a defined P < 0.05. Analyses were carried out using SPSS (version 24) and Microsoft Excel with the Real Statistics Resource Pack software (Release 5.4, copyright 2013-2018; Charles Zaiontz).
Results Demographics Of all the interns, 52.9% (n ¼ 9) responded to the survey. 55.6% of the responders were male (Table 1). 33.3% of the interns (n ¼ 3) were categorical. There was no statistically significant trend in day of week or time relative to work that was used to study; the only “always” responses were recorded when asking if studying occurred at night after working days (Fig. 1). Overall, the most popular times for the interns to study were on days off and at night after working days.
alishahedani et al survey-based surgical curriculum changes
Table 1 e Survey respondent demographics. Respondent characteristic
% (n)
Gender Male
55.6 (5)
Intern training designation Categorical general surgery
33.3 (3)
Noncategorical, designated specialty
22.2 (2)
Noncategorical, nondesignated
44.4 (4)
Voluntary response rate was 52.9%.
Material access In light of the pilot study, the department provided interns a year’s subscription to COACH and SAS. The interns were able to access these websites outside of the curriculum emailed to them by the author team. Already available to the interns were electronic copies of surgical textbooks through SCORE or the University of Florida library system. When considering study materials used, the internet was the most-used resource by the interns (Fig. 2). The internet was also the only resource that did not register any “never” use responses. Review books were the next most-used resource. Within the context of internet resources, there was little variation in what interns used for self-study (Fig. 3), the specifics of which were determined based on free-text responses. The vast majority (66.7%) used SCORE, to which the residency program has had a continuous subscription. SAS, which was subscribed to based on feedback desiring question bank material, was in the minority for independent usage (11.1%). This had similar minimal usage frequency, matching that of Medscape and “Other.” “Other” consisted of urologic sites. There were no freetext responses regarding COACH use. Notably, COACH and SAS were not frequently accessed websites for intern self-study, despite including material that
89
was highlighted as desired in the pilot study. Forty four percentages of the interns responded that they “never” used SAS independently of the links they were emailed by the study team; 55.6% “never” used COACH independently. The highest recorded frequency of independent use of SAS and COACH was “sometimes” (55.6% versus 44.4%). Satisfaction with learning materials in particular was queried (Table 2). Responses leaned toward overall satisfaction compared with dissatisfaction. 66.7% and 44.4% of interns were “somewhat satisfied” with the usefulness of SAS and COACH, respectively, and none were “completely satisfied” with the resources. The COACH logbook for the interns was interrogated and found not to have consistent use throughout the year (Fig. 4). Peak usage was during the first 2 months of the academic year and thereafter dropped. There were no associations of time of day that the site was accessed. There was no increase in usage before the American Board of Surgery In-Training Examination (ABSITE), which is administered at the end of January. In comparison to the pilot study, there was a statistically significant increase in the frequency of material access on the internet (P < 0.05), which was not seen with the other learning materials (Fig. 5). The intern-level teaching curriculum was largely unchanged between the years.
Correlating material use with ABSITE performance In an effort to correlate access of the new online study material with study habit survey responses and ABSITE, one of the initial goals of the study was to use the interns’ deidentification code to cross reference with the access logs of COACH and SAS. However, as the interns did not all identify themselves to be coded consistently throughout the course of the study by the gatekeeper, there was no ability to do this with any statistical vigor. Of the interns who responded, only 4 were able to have their responses and login access correlated. Their ABSITE percentile scores ranged from 30 to 88, with no
Fig. 1 e Timing of interns’ studying based on survey responses. The majority of studying occurred at night, after working during the day. The reported study habits did not correlate with material access, although the data were limited.
90
j o u r n a l o f s u r g i c a l r e s e a r c h o c t o b e r 2 0 1 9 ( 2 4 2 ) 8 7 e9 3
Fig. 2 e Frequency of study materials used by surgical interns. The use of the internet was overall highest, followed by review books. Textbook use was inconsistent. Peer-reviewed journals and personal notes were the least-used for selfstudy.
patterns of access of, or time spent on, the supplemental websites, and no defined trends of time spent studying based on the survey question responses. Categorical interns who were able to be identified from the noncategorical interns appeared to spend more time in self-study, but the small sample size did not allow for meaningful statistical analysis.
Discussion The goal of any residency program is to provide trainees with an educational environment that promotes successful learning; the wealth of study resources available on the internet can both help and hinder efforts to address different methods of learning. By looking at how surgical interns
studied and what they used based on pilot study feedback, this study focused on their resource utilization and satisfaction, which is important to consider when addressing the curriculum for a surgical training program. This study also illustrates that the learner’s interaction with material is critical to curricular success. Based on the pilot study feedback from 2 y prior, subscriptions were purchased for the interns to SAS and COACH. The study focused on the interns, as their weekly educational conference schedule at University of Florida is structured in such a way that allowed for inclusion of these new tools in a nonobtrusive fashion compared with the more senior resident conferences. The supposition that feedback from an intern class only 2 y senior to the study group would be relevant resulted in unexpected observations: the use of the internet
Fig. 3 e Online resource use by surgical interns focused on the websites listed here. SCORE was the most commonly used reported site. “Other” was a urology website.
91
alishahedani et al survey-based surgical curriculum changes
Table 2 e Survey responses when queried about satisfaction of learning resources. Question prompt
Completely dissatisfied % (n)
Somewhat dissatisfied % (n)
Somewhat satisfied % (n)
Completely satisfied % (n)
Did not use/ not applicable % (n)
The study materials you currently use
0
22.2 (2)
66.7 (6)
11.1 (1)
0
The PGY 101 course material
0
22.2 (2)
66.7 (6)
11.1 (1)
0
The ease of use of SCORE
11.1 (1)
33.3 (3)
33.3 (3)
11.1 (1)
11.1 (1)
The usefulness of the COACH surgery education links emailed to you throughout the year
11.1 (1)
33.3 (3)
44.4 (4)
0
11.1 (1)
0
22.2 (2)
66.7 (6)
0
11.1 (1)
The usefulness of the Scientific American Surgery (SAS) readings and quizzes that have been emailed to you weekly
The “Did not use/not applicable” column was not part of the official survey and had been free-texted by interns when they did not use a resource but still recorded a response.
for study increased significantly between the groups, and desired material from the pilot class was not used to the anticipated extent based on pilot study feedback. While this may be attributed to a simple surgical education case of leading a horse to water but not being about to make it drink, it sheds light on the complexities involved in counseling surgical trainees in self-regulated and lifetime learning. Confidence of insight into free-text survey responses is also an important consideration when revising curriculum, as in this study, and will impact future.9 The significant increase in the use of the internet may be attributed to the level of comfort and flexibility the learning platforms offered. While the individuals making up the intern groups were different, their common training level and short time interval between years may reflect a rapid maturation of internet study materials and change in study habits of up-andcoming surgical trainees. The availability of different websites with different features may have inherent broad appeal to a wide range of learning preferences, but the satisfaction responses in this survey suggest otherwise. SCORE, as the most prominent online curriculum for surgical residency programs, continues to mature as its faculty assess content and feedback from program directors.10 It continues to generate readership to a statistically significant extent.4 A high frequency of SCORE use within the category of internet resources was also observed among the surgery interns in this study, who accessed it to a statistically significant greater degree than other internet resources. Analyzing user access of study tools, as was done here and in previous works, can be a useful marker for resource utilization, but it cannot reliably reflect true use.4,11 User logs of most learning platforms, including those used in this study, cannot determine if the trainee read material continuously once logging in or if material was studied with a group of individuals. To this end, this study strove to tease out study habits and determine whether this affected resource utilization. The anticipated implications would be that responses would guide future curricular investments and revisions. The small number of responses did not provide any solid guidance. The prevalence of SCORE as a common study source among trainees regardless of timing before ABSITE examination suggests an extrinsic learning motivation. This has been shown in previous studies related to surgical resident study habits.4,7 But even in one of the most comprehensive study
habit articles to date, data on timing of use of SCORE before ABSITE, particularly whether it was broken down by training year, was not available.7 In addition, it was shown that review books were a largely accessed study source, over SCORE material. If SCORE may be used as a substitute for internet material as a whole, then this present study had the same observations. What remains to be answered is the question of what true overall study resource use is like and how surgical educators can continue to promote resident usage. Extrinsic motivators such as the ABSITE are annual prompts but to tap into a trainee’s intrinsic motivation and help them navigate available material, to use optimally, is the goal. The trend noted in this study’s results of categorical interns committing time to self-study can help to frame expectations for all interns when providing them with resources. Interns are novice learners when considered alongside senior residents and attending surgeons. Although the purpose of the study was to use the pilot study feedback to provide more user-friendly and quality learning materials, the amount of material provided to the interns may have been overwhelming. Our results indicate that the new online resources were most accessed when they were first introduced. Resource use did not increase before the ABSITE examination. We postulate that this was due to undefined resource use guidelines and resource fatigue (being overwhelmed and unable to prioritize and use study material effectively, often in the setting of being offered many items from which to study). This may have also contributed to the interns’ satisfaction responses with the new material. As has been discussed in other medical professions, it may also be inferred that novice surgical interns, unlike residents and fellows, approach learning about a problem with less background and possibly less intention and inner direction.12 More simply put, the novice learner here may not know “where to start.” As a result, their access to multiple online platforms can be overwhelming and reduce their motivation. Notably, access to the new tools was high at start of the academic year but quickly diminished. If the application of new learning material was more focused, the overall use of new material may have increased. The novice learner does not always know what they need and how to retrieve the information, which can lead to learner fatigue, making guided and selfregulated learning all the more halting. Research in nonmedical higher education, looking at online environments, has
92
j o u r n a l o f s u r g i c a l r e s e a r c h o c t o b e r 2 0 1 9 ( 2 4 2 ) 8 7 e9 3
Fig. 4 e Access to COACH, a predominantly video- and animation-based website, was not consistently used by interns, despite regular emails during the academic year with links directly to specific material that aligned with the TWIS curriculum. There was no increase in use closer to January, the month of the annual in-service examination.
shown how information overload (cognitive overload) can affect learning.13,14 A major limitation of this study was the small sample size and the inability to fully code survey responses, to correlate individual responses with website access and ABSITE score as a way to objectively measure resource use outcomes. Although the ABSITE does not reflect the full picture of a trainee’s performance, it does provide a measurable data point that has been shown to be associated with board passage success.15 Even with the small group, more complete data may have provided more substantive basis for making recommendations for individuals or level-specific curriculum revisions. Furthermore, because the minority of interns were deidentified, we could not correlate the use of material and the ABSITE scores. Perhaps, the interns who sought out and used the provided material are individuals who would
perform well on ABSITE regardless. Long-term efforts to expand this to multiple classes and multiple institutions would allow for more vigorous analysis. Another limitation was the method by which the study resources were assessed. Survey feedback was used to create a curriculum for the learner. Creating a curriculum that the learner will use is clearly an educator’s goal. As curricular enhancements here were created based on intern feedback, the anticipated outcome was that these resources would be used with a degree of consistency. However, our survey results clearly show that despite the surveyed intern classes both being contemporary to each other, what one group desired to use, the other did not. To implement a better surgical educational curriculum, emphasis could be placed on learner-chosen topics, which has been shown to improve mastery and active learning in the
Fig. 5 e When comparing internet use by surveyed year, there was a statistically significant difference in response trends between years; the sample size was overall too small to comment on stat significance by frequency group. (P < 0.05).
alishahedani et al survey-based surgical curriculum changes
nonmedical setting.16 However, it has also been discussed that placing learning decisions too much in the hands of the student may be deleterious to their knowledge base.17 Within the setting of medical education, mentored studying guided by a senior resident or faculty member would be a way to counsel the novice intern learner. Focusing conference teaching by training level and using online multimedia has been shown in previous studies to improve fund of knowledge and examination performance in students and residents.18,19 Creating a focused structure for the more junior surgical trainees would provide them with the background to navigate the abundance of educational material now available in all forms of media. In addition, having organized learning material that may be accessed by the resident outside of a training program’s defined curriculum can create an environment of asynchronous education that can promote successful independent learning.20 Our survey and its follow-up are evidence that surgical trainees have a sense of what would help their learning but that they would likely benefit from purposeful guidance from educators.
Conclusions The ideal curriculum for surgical trainees should be focused and organized; this is paramount when considering the intern learner. Multiple resources may prove inefficient and promote resource fatigue. Interval review of use based on training level may help keep program curricula comprehensive, while keeping them relevant and highly accessed. Implementing a consistent curriculum in study resources can improve objective measures and provide better skills and knowledge for tomorrow’s surgeon.
Acknowledgment The authors would like to thank Mr. Jim Mullins for providing administrative support and deidentification services for the study survey. The authors extend their gratitude to the University of Florida, Department of Surgery, for funding this study through the Faculty Career Development Grant. Authors’ contributions: M.E.A. organized the survey data, performed statistics, interpreted data, and primarily authored this article. G.S.A. critically reviewed the article and its revisions. J.A.T. conceived the project idea, reviewed and performed statistical analysis, and critically reviewed and edited this article.
Disclosure The authors report no proprietary or commercial interest in any product mentioned or concept discussed in this article.
Supplementary data Supplementary data related to this article can be found at https://doi.org/10.1016/j.jss.2019.04.044.
93
references
1. Sachdeva AK, Bell RH, Britt LD, et al. National efforts to reform residency education in surgery. Acad Med. 2007;82:1200e1210. 2. Moalem J, Edhayan E, DaRosa DA, et al. Incorporating the SCORE curriculum and web site into your residency. J Surg Educ. 2011;68:294e297. 3. Goldstein SD, Papandria D, Linden A, et al. A pilot comparison of standardized online surgical curricula for use in low- and middle-income Countries. JAMA Surg. 2014;149:341e346. 4. Joshi ART, Salami A, Hickey M, et al. What can SCORE web portal usage analytics tell us about how surgical residents learn? J Surg Educ. 2017;74:e133ee137. 5. Purcell GP. Surgical textbooks: past, present, and future. Ann Surg. 2003;238:S34eS41. 6. Yeh DD, Hwabejire JO, Imam A, et al. A survey of study habits of general surgery residents. J Surg Educ. 2013;70:15e23. 7. Kim JJ, Kim DY, Kaji AH, et al. Reading habits of general surgery residents and association with American board of surgery in-training examination performance. JAMA Surg. 2015;150:882e889. 8. Taylor JA, Shaw CM, Tan SA, et al. Are the kids alright? Review books and the internet as the most common study resources for the general surgery clerkship. Am J Surg. 2018;215:191e195. 9. LaDonna KA, Taylor T, Lingard L. Why open-ended survey questions are unlikely to support rigorous qualitative insights. Acad Med. 2018;93:347e349. 10. Klingensmith ME, Jones AT, Smiley W, et al. Subscription to the surgical council on resident education web portal and qualifying examination performance. J Am Coll Surg. 2014;218:566e570. 11. O’Leary DP, Corrigan MA, McHugh SM, et al. From theater to the world wide web–a new online era for surgical education. J Surg Educ. 2012;69:483e486. 12. Persky AM, Robinson JD. Moving from novice to expertise and its implications for instruction. Am J Pharm Educ. 2017;81:6065. 13. Chen C, Pederson S, Murphy KL. Learners’ perceived information overload in online learning via computermediated communication. Res Learn Technol. 2011;19:101e116. 14. Blayney P, Kalyuga S, Sweller J. Using cognitive load theory to tailor instruction to levels of accounting students’ expertise. Educ Tech Soc. 2015;18:199e210. 15. Taggarshe D, Mittal V. Improving surgical resident’s performance in the American Board of Surgery in Training Examination (ABSITE)–do review courses help? The program directors’ perspective. J Surg Educ. 2011;68:24e28. 16. Hanewicz C, Platt A, Arendt A. Creating a learner-centered teaching environment using student choice in assignments. Distance Education. 2017;38:273e287. 17. Kirschner PA, van Merrienboer JJG. Do learners really know best? Urban legends in education. Educ Pscyhol. 2013;48:169e183. 18. Ridgway PF, Sheikh A, Sweeney KJ, et al. Surgical e-learning: validation of multimedia web-based lectures. Med Educ. 2007;41:168e172. 19. Farrohki ET, Jensen AR, Brock DM, et al. Expanding resident conferences while tailoring them to level of training: a longitudinal Study. J Surg Educ. 2008;65:84e90. 20. Mallin M, Schlein S, Doctor S, et al. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014;89:598e601.
Available online at www.sciencedirect.com
ScienceDirect journal homepage: www.JournalofSurgicalResearch.com
Implementing Survey-Based Changes and Analyzing Usage in an Online Curriculum for Surgical Interns Mohammadali E. Alishahedani, BA, George A. Sarosi, MD, and Janice A. Taylor, MD, MEd* Department of Surgery, University of Florida College of Medicine, Gainesville Florida
article info
abstract
Article history:
Background: To improve the online curriculum at the authors’ program, study habits and
Received 13 November 2018
learning resources of surgical interns were evaluated. Based on the results, changes were
Received in revised form
implemented to align with their teaching conferences. We sought to determine utilization
2 April 2019
and satisfaction with the new materials.
Accepted 9 April 2019
Methods: At the end of the 2017 academic year, surgical interns at a single institution
Available online xxx
voluntarily responded to an Institutional Review Boardeapproved survey regarding new and established learning material use and study habits. Responses were deidentified.
Keywords:
Descriptive statistics were performed on demographics. Likert responses underwent Mann
Surgical education
eWhitney analysis (a ¼ 0.05).
Online learning
Results: The response rate was 52.9% (n ¼ 9). The internet was the most-used resource
Study habits
(P < 0.05). All respondents used the internet to varying degrees. Textbooks and the internet were always used by 22.2%. 33.3% never used textbooks to study. There was a statistically significant increase in internet access but not with other materials (P < 0.05). Regarding new material organized from the pilot, none was consistently accessed. 55.6% sometimes used new weekly reading links associated with a question bank. 66.7% were somewhat satisfied with these links. 44.4% were somewhat satisfied with new video links, organized in parallel with This Week in Surgical Council on Resident Education. Limited sample size did not allow for meaningful statistical analysis of material use with American Board of Surgery In-Training Examination scores. Conclusions: Overall, materials organized based on pilot study feedback were not widely used, and satisfaction was limited. The statistically significant increase in use of internet resources warrants attention. Focusing changes in future curricular design may help create a more effective learning environment. ª 2019 Elsevier Inc. All rights reserved.
Introduction The role of the internet is increasing in surgical education.1-4 Traditional resources, such as “classic” textbooks like Schwartz’s Principles of Surgery and Greenfield’s Surgery,
while still widely used and referenced to build one’s knowledge base, are joined by their electronic counterparts. Internet-based learning platformsdvideo sites, interactive question banks, and customizable reading schedulesdadd to the figurative library of the surgical trainee. Previously, Purcell
* Corresponding author. Department of Surgery, University of Florida College of Medicine, 1600 SW Archer Rd, PO Box 100119, Gainesville FL 32610. Tel.: þ15135828316; fax: þ1 352 273 8772. E-mail address: [email protected] (J.A. Taylor). 0022-4804/$ e see front matter ª 2019 Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jss.2019.04.044
88
j o u r n a l o f s u r g i c a l r e s e a r c h o c t o b e r 2 0 1 9 ( 2 4 2 ) 8 7 e9 3
outlined the evolution of technology in learning resources.5 As surgical trainees use materials outside of their institutional curriculum, there is not only a need to understand their access and study habits but also a need to offer appropriate resources that will be used regularly.4,6,7 One widely known incorporation of internet-based learning into an organized, standardized curriculum is Surgical Council on Resident Education (SCORE), the program put forth by the SCORE.1 Since its inception, SCORE has integrated various learning tools to foster a better learning environment for the surgical trainee. In 2015, SCORE introduced This Week in Score, an organized 2-y curriculum for surgical residents, which acts as a guide through the site’s material for the user.4 Although SCORE is the recognized standard among online surgical curricula, and is the backbone of the authors’ residency teaching schedule, it is used along with other material based on the residents’ study preferences, access capabilities, and individual institutional account. To this end, at the authors’ institution, there are multiple online resources provided to the surgical interns, which include e-book and peerreviewed journal access provided by the university. In an effort to improve the didactic experience of the surgical interns at the authors’ institution, intern classes were surveyed at the end of the 2015 and 2017 academic years. Resource utilization and satisfaction were analyzed in the 2015 pilot study. Based on the results, changes were made to the interns’ organized teaching curriculum. The intent was to introduce new online study tools that would correlate with what interns stated they would use, or desired access to, to give access to different media- or problem-based content to promote more efficient learning, more consistent reading, and increase access to quality just-in-time learning. Then, a follow-up survey was performed in 2017. We expected an increase in intern satisfaction and resource utilization. In addition, we hypothesized that this would have a positive effect on objective markers such as the annual in-service examination.
Material and methods Educational materials Based on 2015 pilot study feedback regarding study material use, 1-y subscriptions were purchased to Scientific American Surgery (SAS; https://www.deckerip.com/products/scientificamerican-surgery/) and COACH Surgery (COACH; https:// collectedmed.com/coachsurgery). SAS provided an extensive question bank and problem-based learning formats; COACH provided multimedia content such as curated video and animation links. The decision to subscribe to these sites was made with input from the Executive Education Committee of the University of Florida, Department of Surgery. For 1 y, interns were sent weekly links from SAS, with questions, based on their department educational conference topics. On average, links from COACH were emailed every other week, based on the This Week in SCORE curriculum used by the interns. The interns had general access to COACH and SAS for self-studying purposes. The departmental educational website also had high-yield links to SCORE (https://www.
surgicalcore.org/), organized based on the weekly conference schedule. Where available, website access histories were used to retrieve data regarding intern activity. For example, both COACH and SAS tracked the number of logins and dates, and which area of the sites were accessed, but COACH also tracked the time of day and length of time for each login. SCORE logged date, time, and visit duration of each access to the site, but access to the high-yield links from SCORE that were added the department educational website were not trackable.
Survey tool and recruitment To evaluate utilization and satisfaction of the new curriculum, a survey tool was used, exempted by the University of Florida Institutional Review Board. The 20-question survey was administered to surgical interns at the University of Florida in Gainesville, at the end of the 2016-2017 academic year (Appendix). The survey was verified by a modified Delphi technique and was based on previously published surveys.6,8 Recruitment for the survey was based on participation during one of the residents’ weekly mandatory general meetings in June. The study team presented the survey tool and study purpose. Participation was voluntary. The survey was administered in pencil/paper fashion. Those who participated were not required to write their name on the survey. There were a total of 17 interns in the program. The study team left the meeting room during completion of the survey by the interns, to minimize any perceived outside influence on responses. It was collected by the designated study gatekeeper, who also deidentified the results.
Statistical analysis In responding to the survey, the interns were asked to consider different aspects of their study habits and resource use. Responses were recorded based on multiple choice, Likert scale, and free text. When interns completed the name field, the study gatekeeper was able to correlate their deidentification code with in-service performance and access history to the online resources, COACH and SAS. Descriptive statistics were performed on the demographic data. KruskaleWallis and ManneWhitney analyses were performed on the Likert responses. Statistics were performed using a defined P < 0.05. Analyses were carried out using SPSS (version 24) and Microsoft Excel with the Real Statistics Resource Pack software (Release 5.4, copyright 2013-2018; Charles Zaiontz).
Results Demographics Of all the interns, 52.9% (n ¼ 9) responded to the survey. 55.6% of the responders were male (Table 1). 33.3% of the interns (n ¼ 3) were categorical. There was no statistically significant trend in day of week or time relative to work that was used to study; the only “always” responses were recorded when asking if studying occurred at night after working days (Fig. 1). Overall, the most popular times for the interns to study were on days off and at night after working days.
alishahedani et al survey-based surgical curriculum changes
Table 1 e Survey respondent demographics. Respondent characteristic
% (n)
Gender Male
55.6 (5)
Intern training designation Categorical general surgery
33.3 (3)
Noncategorical, designated specialty
22.2 (2)
Noncategorical, nondesignated
44.4 (4)
Voluntary response rate was 52.9%.
Material access In light of the pilot study, the department provided interns a year’s subscription to COACH and SAS. The interns were able to access these websites outside of the curriculum emailed to them by the author team. Already available to the interns were electronic copies of surgical textbooks through SCORE or the University of Florida library system. When considering study materials used, the internet was the most-used resource by the interns (Fig. 2). The internet was also the only resource that did not register any “never” use responses. Review books were the next most-used resource. Within the context of internet resources, there was little variation in what interns used for self-study (Fig. 3), the specifics of which were determined based on free-text responses. The vast majority (66.7%) used SCORE, to which the residency program has had a continuous subscription. SAS, which was subscribed to based on feedback desiring question bank material, was in the minority for independent usage (11.1%). This had similar minimal usage frequency, matching that of Medscape and “Other.” “Other” consisted of urologic sites. There were no freetext responses regarding COACH use. Notably, COACH and SAS were not frequently accessed websites for intern self-study, despite including material that
89
was highlighted as desired in the pilot study. Forty four percentages of the interns responded that they “never” used SAS independently of the links they were emailed by the study team; 55.6% “never” used COACH independently. The highest recorded frequency of independent use of SAS and COACH was “sometimes” (55.6% versus 44.4%). Satisfaction with learning materials in particular was queried (Table 2). Responses leaned toward overall satisfaction compared with dissatisfaction. 66.7% and 44.4% of interns were “somewhat satisfied” with the usefulness of SAS and COACH, respectively, and none were “completely satisfied” with the resources. The COACH logbook for the interns was interrogated and found not to have consistent use throughout the year (Fig. 4). Peak usage was during the first 2 months of the academic year and thereafter dropped. There were no associations of time of day that the site was accessed. There was no increase in usage before the American Board of Surgery In-Training Examination (ABSITE), which is administered at the end of January. In comparison to the pilot study, there was a statistically significant increase in the frequency of material access on the internet (P < 0.05), which was not seen with the other learning materials (Fig. 5). The intern-level teaching curriculum was largely unchanged between the years.
Correlating material use with ABSITE performance In an effort to correlate access of the new online study material with study habit survey responses and ABSITE, one of the initial goals of the study was to use the interns’ deidentification code to cross reference with the access logs of COACH and SAS. However, as the interns did not all identify themselves to be coded consistently throughout the course of the study by the gatekeeper, there was no ability to do this with any statistical vigor. Of the interns who responded, only 4 were able to have their responses and login access correlated. Their ABSITE percentile scores ranged from 30 to 88, with no
Fig. 1 e Timing of interns’ studying based on survey responses. The majority of studying occurred at night, after working during the day. The reported study habits did not correlate with material access, although the data were limited.
90
j o u r n a l o f s u r g i c a l r e s e a r c h o c t o b e r 2 0 1 9 ( 2 4 2 ) 8 7 e9 3
Fig. 2 e Frequency of study materials used by surgical interns. The use of the internet was overall highest, followed by review books. Textbook use was inconsistent. Peer-reviewed journals and personal notes were the least-used for selfstudy.
patterns of access of, or time spent on, the supplemental websites, and no defined trends of time spent studying based on the survey question responses. Categorical interns who were able to be identified from the noncategorical interns appeared to spend more time in self-study, but the small sample size did not allow for meaningful statistical analysis.
Discussion The goal of any residency program is to provide trainees with an educational environment that promotes successful learning; the wealth of study resources available on the internet can both help and hinder efforts to address different methods of learning. By looking at how surgical interns
studied and what they used based on pilot study feedback, this study focused on their resource utilization and satisfaction, which is important to consider when addressing the curriculum for a surgical training program. This study also illustrates that the learner’s interaction with material is critical to curricular success. Based on the pilot study feedback from 2 y prior, subscriptions were purchased for the interns to SAS and COACH. The study focused on the interns, as their weekly educational conference schedule at University of Florida is structured in such a way that allowed for inclusion of these new tools in a nonobtrusive fashion compared with the more senior resident conferences. The supposition that feedback from an intern class only 2 y senior to the study group would be relevant resulted in unexpected observations: the use of the internet
Fig. 3 e Online resource use by surgical interns focused on the websites listed here. SCORE was the most commonly used reported site. “Other” was a urology website.
91
alishahedani et al survey-based surgical curriculum changes
Table 2 e Survey responses when queried about satisfaction of learning resources. Question prompt
Completely dissatisfied % (n)
Somewhat dissatisfied % (n)
Somewhat satisfied % (n)
Completely satisfied % (n)
Did not use/ not applicable % (n)
The study materials you currently use
0
22.2 (2)
66.7 (6)
11.1 (1)
0
The PGY 101 course material
0
22.2 (2)
66.7 (6)
11.1 (1)
0
The ease of use of SCORE
11.1 (1)
33.3 (3)
33.3 (3)
11.1 (1)
11.1 (1)
The usefulness of the COACH surgery education links emailed to you throughout the year
11.1 (1)
33.3 (3)
44.4 (4)
0
11.1 (1)
0
22.2 (2)
66.7 (6)
0
11.1 (1)
The usefulness of the Scientific American Surgery (SAS) readings and quizzes that have been emailed to you weekly
The “Did not use/not applicable” column was not part of the official survey and had been free-texted by interns when they did not use a resource but still recorded a response.
for study increased significantly between the groups, and desired material from the pilot class was not used to the anticipated extent based on pilot study feedback. While this may be attributed to a simple surgical education case of leading a horse to water but not being about to make it drink, it sheds light on the complexities involved in counseling surgical trainees in self-regulated and lifetime learning. Confidence of insight into free-text survey responses is also an important consideration when revising curriculum, as in this study, and will impact future.9 The significant increase in the use of the internet may be attributed to the level of comfort and flexibility the learning platforms offered. While the individuals making up the intern groups were different, their common training level and short time interval between years may reflect a rapid maturation of internet study materials and change in study habits of up-andcoming surgical trainees. The availability of different websites with different features may have inherent broad appeal to a wide range of learning preferences, but the satisfaction responses in this survey suggest otherwise. SCORE, as the most prominent online curriculum for surgical residency programs, continues to mature as its faculty assess content and feedback from program directors.10 It continues to generate readership to a statistically significant extent.4 A high frequency of SCORE use within the category of internet resources was also observed among the surgery interns in this study, who accessed it to a statistically significant greater degree than other internet resources. Analyzing user access of study tools, as was done here and in previous works, can be a useful marker for resource utilization, but it cannot reliably reflect true use.4,11 User logs of most learning platforms, including those used in this study, cannot determine if the trainee read material continuously once logging in or if material was studied with a group of individuals. To this end, this study strove to tease out study habits and determine whether this affected resource utilization. The anticipated implications would be that responses would guide future curricular investments and revisions. The small number of responses did not provide any solid guidance. The prevalence of SCORE as a common study source among trainees regardless of timing before ABSITE examination suggests an extrinsic learning motivation. This has been shown in previous studies related to surgical resident study habits.4,7 But even in one of the most comprehensive study
habit articles to date, data on timing of use of SCORE before ABSITE, particularly whether it was broken down by training year, was not available.7 In addition, it was shown that review books were a largely accessed study source, over SCORE material. If SCORE may be used as a substitute for internet material as a whole, then this present study had the same observations. What remains to be answered is the question of what true overall study resource use is like and how surgical educators can continue to promote resident usage. Extrinsic motivators such as the ABSITE are annual prompts but to tap into a trainee’s intrinsic motivation and help them navigate available material, to use optimally, is the goal. The trend noted in this study’s results of categorical interns committing time to self-study can help to frame expectations for all interns when providing them with resources. Interns are novice learners when considered alongside senior residents and attending surgeons. Although the purpose of the study was to use the pilot study feedback to provide more user-friendly and quality learning materials, the amount of material provided to the interns may have been overwhelming. Our results indicate that the new online resources were most accessed when they were first introduced. Resource use did not increase before the ABSITE examination. We postulate that this was due to undefined resource use guidelines and resource fatigue (being overwhelmed and unable to prioritize and use study material effectively, often in the setting of being offered many items from which to study). This may have also contributed to the interns’ satisfaction responses with the new material. As has been discussed in other medical professions, it may also be inferred that novice surgical interns, unlike residents and fellows, approach learning about a problem with less background and possibly less intention and inner direction.12 More simply put, the novice learner here may not know “where to start.” As a result, their access to multiple online platforms can be overwhelming and reduce their motivation. Notably, access to the new tools was high at start of the academic year but quickly diminished. If the application of new learning material was more focused, the overall use of new material may have increased. The novice learner does not always know what they need and how to retrieve the information, which can lead to learner fatigue, making guided and selfregulated learning all the more halting. Research in nonmedical higher education, looking at online environments, has
92
j o u r n a l o f s u r g i c a l r e s e a r c h o c t o b e r 2 0 1 9 ( 2 4 2 ) 8 7 e9 3
Fig. 4 e Access to COACH, a predominantly video- and animation-based website, was not consistently used by interns, despite regular emails during the academic year with links directly to specific material that aligned with the TWIS curriculum. There was no increase in use closer to January, the month of the annual in-service examination.
shown how information overload (cognitive overload) can affect learning.13,14 A major limitation of this study was the small sample size and the inability to fully code survey responses, to correlate individual responses with website access and ABSITE score as a way to objectively measure resource use outcomes. Although the ABSITE does not reflect the full picture of a trainee’s performance, it does provide a measurable data point that has been shown to be associated with board passage success.15 Even with the small group, more complete data may have provided more substantive basis for making recommendations for individuals or level-specific curriculum revisions. Furthermore, because the minority of interns were deidentified, we could not correlate the use of material and the ABSITE scores. Perhaps, the interns who sought out and used the provided material are individuals who would
perform well on ABSITE regardless. Long-term efforts to expand this to multiple classes and multiple institutions would allow for more vigorous analysis. Another limitation was the method by which the study resources were assessed. Survey feedback was used to create a curriculum for the learner. Creating a curriculum that the learner will use is clearly an educator’s goal. As curricular enhancements here were created based on intern feedback, the anticipated outcome was that these resources would be used with a degree of consistency. However, our survey results clearly show that despite the surveyed intern classes both being contemporary to each other, what one group desired to use, the other did not. To implement a better surgical educational curriculum, emphasis could be placed on learner-chosen topics, which has been shown to improve mastery and active learning in the
Fig. 5 e When comparing internet use by surveyed year, there was a statistically significant difference in response trends between years; the sample size was overall too small to comment on stat significance by frequency group. (P < 0.05).
alishahedani et al survey-based surgical curriculum changes
nonmedical setting.16 However, it has also been discussed that placing learning decisions too much in the hands of the student may be deleterious to their knowledge base.17 Within the setting of medical education, mentored studying guided by a senior resident or faculty member would be a way to counsel the novice intern learner. Focusing conference teaching by training level and using online multimedia has been shown in previous studies to improve fund of knowledge and examination performance in students and residents.18,19 Creating a focused structure for the more junior surgical trainees would provide them with the background to navigate the abundance of educational material now available in all forms of media. In addition, having organized learning material that may be accessed by the resident outside of a training program’s defined curriculum can create an environment of asynchronous education that can promote successful independent learning.20 Our survey and its follow-up are evidence that surgical trainees have a sense of what would help their learning but that they would likely benefit from purposeful guidance from educators.
Conclusions The ideal curriculum for surgical trainees should be focused and organized; this is paramount when considering the intern learner. Multiple resources may prove inefficient and promote resource fatigue. Interval review of use based on training level may help keep program curricula comprehensive, while keeping them relevant and highly accessed. Implementing a consistent curriculum in study resources can improve objective measures and provide better skills and knowledge for tomorrow’s surgeon.
Acknowledgment The authors would like to thank Mr. Jim Mullins for providing administrative support and deidentification services for the study survey. The authors extend their gratitude to the University of Florida, Department of Surgery, for funding this study through the Faculty Career Development Grant. Authors’ contributions: M.E.A. organized the survey data, performed statistics, interpreted data, and primarily authored this article. G.S.A. critically reviewed the article and its revisions. J.A.T. conceived the project idea, reviewed and performed statistical analysis, and critically reviewed and edited this article.
Disclosure The authors report no proprietary or commercial interest in any product mentioned or concept discussed in this article.
Supplementary data Supplementary data related to this article can be found at https://doi.org/10.1016/j.jss.2019.04.044.
93
references
1. Sachdeva AK, Bell RH, Britt LD, et al. National efforts to reform residency education in surgery. Acad Med. 2007;82:1200e1210. 2. Moalem J, Edhayan E, DaRosa DA, et al. Incorporating the SCORE curriculum and web site into your residency. J Surg Educ. 2011;68:294e297. 3. Goldstein SD, Papandria D, Linden A, et al. A pilot comparison of standardized online surgical curricula for use in low- and middle-income Countries. JAMA Surg. 2014;149:341e346. 4. Joshi ART, Salami A, Hickey M, et al. What can SCORE web portal usage analytics tell us about how surgical residents learn? J Surg Educ. 2017;74:e133ee137. 5. Purcell GP. Surgical textbooks: past, present, and future. Ann Surg. 2003;238:S34eS41. 6. Yeh DD, Hwabejire JO, Imam A, et al. A survey of study habits of general surgery residents. J Surg Educ. 2013;70:15e23. 7. Kim JJ, Kim DY, Kaji AH, et al. Reading habits of general surgery residents and association with American board of surgery in-training examination performance. JAMA Surg. 2015;150:882e889. 8. Taylor JA, Shaw CM, Tan SA, et al. Are the kids alright? Review books and the internet as the most common study resources for the general surgery clerkship. Am J Surg. 2018;215:191e195. 9. LaDonna KA, Taylor T, Lingard L. Why open-ended survey questions are unlikely to support rigorous qualitative insights. Acad Med. 2018;93:347e349. 10. Klingensmith ME, Jones AT, Smiley W, et al. Subscription to the surgical council on resident education web portal and qualifying examination performance. J Am Coll Surg. 2014;218:566e570. 11. O’Leary DP, Corrigan MA, McHugh SM, et al. From theater to the world wide web–a new online era for surgical education. J Surg Educ. 2012;69:483e486. 12. Persky AM, Robinson JD. Moving from novice to expertise and its implications for instruction. Am J Pharm Educ. 2017;81:6065. 13. Chen C, Pederson S, Murphy KL. Learners’ perceived information overload in online learning via computermediated communication. Res Learn Technol. 2011;19:101e116. 14. Blayney P, Kalyuga S, Sweller J. Using cognitive load theory to tailor instruction to levels of accounting students’ expertise. Educ Tech Soc. 2015;18:199e210. 15. Taggarshe D, Mittal V. Improving surgical resident’s performance in the American Board of Surgery in Training Examination (ABSITE)–do review courses help? The program directors’ perspective. J Surg Educ. 2011;68:24e28. 16. Hanewicz C, Platt A, Arendt A. Creating a learner-centered teaching environment using student choice in assignments. Distance Education. 2017;38:273e287. 17. Kirschner PA, van Merrienboer JJG. Do learners really know best? Urban legends in education. Educ Pscyhol. 2013;48:169e183. 18. Ridgway PF, Sheikh A, Sweeney KJ, et al. Surgical e-learning: validation of multimedia web-based lectures. Med Educ. 2007;41:168e172. 19. Farrohki ET, Jensen AR, Brock DM, et al. Expanding resident conferences while tailoring them to level of training: a longitudinal Study. J Surg Educ. 2008;65:84e90. 20. Mallin M, Schlein S, Doctor S, et al. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014;89:598e601.