- Email: [email protected]
What Can SCORE Web Portal Usage Analytics Tell Us About How Surgical Residents Learn?
2017 APDS SPRING MEETING
What Can SCORE Web Portal Usage Analytics Tell Us About How Surgical Residents Learn? Amit R.T. Joshi, MD, FACS,*,1 Aitua Salami, MD,* Mark Hickey, MA,†,2 Kerry B. Barrett, BA,†,2 Mary E. Klingensmith, MD, FACS,‡,1 and Mark A. Malangoni, MD, FACS§,1 Department of Surgery, Einstein Healthcare Network, Philadelphia, Pennsylvania; †Surgical Council on Resident Education, Philadelphia, Pennsylvania; ‡Department of Surgery, Washington University, St. Louis, Missouri; and §American Board of Surgery, Philadelphia, Pennsylvania *
PURPOSE: The Surgical Council on Resident Education
(SCORE) was established in 2004 with 2 goals: to develop a standardized, competency-based curriculum for general surgery residency training; and to develop a web portal to deliver this content. By 2012, 96% of general surgery residency programs subscribed to the SCORE web portal. Surgical educators have previously described the myriad ways they have incorporated SCORE into their curricula. The aim of this study was to analyze user data to describe how and when residents use SCORE. METHODS: Using analytic software, we measured SCORE usage from July, 2013 to June, 2016. Data such as IP addresses, geo-tagging coordinates, and operating system platforms were collected. The primary outcome was the median duration of SCORE use. Secondary outcomes were the time of day and the operating system used when accessing SCORE. Descriptive statistics were performed, and a p o 0.05 was deemed statistically significant.
(14.2) compared to July to October and March to June (12.3 [IQR: 3.2] and 9.6 minute/session [IQR: 2.2]), respectively (p o 0.001). This seasonal trend continued for all 3 years. We observed an increased number of sessions per day over the 3 years: median of 1500 sessions/d (IQR: 1115) vs 1706 (IQR: 1334) vs 1728 (IQR: 1352), p o 0.001. (Fig.). Most SCORE sessions occurred at night: 38,011 (IQR: 4532) vs 17,529 (IQR: 19,850) during the day (p o 0.001). Windows was the most frequently used operating system at 48.9% (p o 0.001 vs others). CONCLUSIONS: SCORE usage has increased significantly
over the last 3 years, when measured by number of sessions per day and length of time per session. There are predictable daily, diurnal, and seasonal variations in SCORE usage. The annual in-training examination is a prominent factor C 2017 stimulating SCORE usage. ( J Surg Ed ]:]]]-]]]. J Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
RESULTS: There were 42,743 total SCORE subscribers
KEY WORDS: surgery residents, study habits, ABSITE,
during the study period (75% resident and 25% faculty) with a mean of 14,248 subscribers per year. The overall median duration of SCORE use was 11.9 minute/session (interquartile range [IQR]: 6.8). Additionally, there was a significant increase in session length over the 3 academic years; 10.1 (IQR: 6.4), 11.9 (IQR: 7.2), and 13.2 minute/ session (IQR: 5.4) in 2013 to 2014, 2014 to 2015, and 2015 to 2016, respectively (p o 0.001). SCORE usage was highest in November to February at 21.0 minute/session
SCORE
This paper was presented as an oral abstract at the annual meeting of the APDS, San Diego, 2017. Correspondence: Inquiries to Amit R.T. Joshi, MD, FACS, Department of Surgery, Einstein Healthcare Network, 5401 Old York Road, Suite 510, Philadelphia, PA 19141; fax: þ(215)-663-6443; e-mail: [email protected] 1
Serve as consultants to SCORE.
2
Employees of SCORE.
COMPETENCIES: Medical Knowledge, Practice based Learning and Improvement, Professionalism
INTRODUCTION The Surgical Council on Resident Education (SCORE) was established in 2004 by a consortium of surgical organizations: the American Board of Surgery (ABS), American College of Surgeons, American Surgical Association, Association of Program Directors in Surgery, Association for Surgical Education, Accreditation Council for Graduate Medical Education (ACGME), and Society of American Gastrointestinal and Endoscopic Surgeons.1 SCORE developed a comprehensive, competency-based curriculum for general surgery residency training available through an
Journal of Surgical Education & 2017 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2017.10.002
1
online portal.2 Currently, 97% of the 272 ACGME-accredited general surgery residency training programs subscribe to SCORE. The SCORE web portal was launched in 2009 as a learning management system for general surgery residency programs and their trainees to use as a resource for study and education. The content in SCORE is organized around 750 discrete modules, which contain learning objectives, study questions, text and multimedia resources, and self-assessment questions. Since 2015, the SCORE web portal has provided an additional organizational framework whereby users and programs can cycle through the entire SCORE content in a defined 2-year curriculum. This sequence, This Week in SCORE (TWIS), includes a cluster of related modules paired with 10 peer-reviewed self-assessment questions to aid in preparation for each week’s educational content. Trainees and programs are able to track their performance on these weekly quizzes. TWIS is also used by many programs as the basis for didactic conferences. Investigators have previously studied correlations between in-training examinations and board certification.3-5 Chang et al.6 examined the sources used by residents. Prior work has demonstrated a considerable improvement in mean ABS qualifying examination scaled scores for residents in programs that initially subscribed to the SCORE web portal.7 The purpose of this report was to analyze SCORE portal usage data to determine how residents study on a daily and weekly basis and if patterns change during the academic year.
METHODS We used Google Analytics (Mountain View, CA) to measure SCORE portal use from July, 2013 to June, 2016. Once a
user has logged onto the SCORE portal, that user’s internet protocol (IP) address can generate an approximate geographic location. Data such as geo-tagging coordinates, session length, and operating system platforms were collected, in concordance with each user′s terms of service agreement that allows the collection of “aggregated, anonymous statistical information about use of the portal by licensees.” IP addresses were not linked with individual users. The primary outcome was the median duration of SCORE portal use as measured by logged-in time. Secondary outcomes were the time of day for logins and the operating system used for access. Descriptive statistics were performed, and a p o 0.05 was deemed statistically significant.
RESULTS There were 42,743 individual subscribers to the SCORE website during the 3-year study period. Subscribers comprised 75% resident and 25% faculty, with a mean of 14,248 subscribers per year from and a mean of 433 residency programs per year. IP address tagging indicated that subscribers overwhelmingly resided in the United States. There were, however, subscribers from every inhabited continent (Fig. 1). The overall median duration of SCORE use was 11.9 minute/session (interquartile range [IQR]: 6.8). Additionally, there was a significant increase in session duration over the 3 academic years; 10.1 (IQR: 6.4), 11.9 (IQR: 7.2), and 13.2 minute/session (IQR: 5.4) in 2013 to 2014, 2014 to 2015, and 2015 to 2016, respectively (p o 0.001). SCORE portal usage was highest from November to February at 21.0 minute/session (14.2) compared to July
FIGURE 1. Countries where the SCORE portal was accessed from 2013 to 2016. 2
Journal of Surgical Education Volume ]/Number ] ] 2017
2013-2014
2014-2015
2015-2016
60 50 40 30 20 10
Jul Jul Jul Aug Aug Sep Sep Sep Oct Oct Nov Nov Dec Dec Dec Jan Jan Feb Feb Mar Mar Mar Apr Apr May May Jun Jun Jun
0
FIGURE 2. Mean session length (min).
to October and March to June (12.3 [IQR: 3.2] and 9.6 minute/session [IQR: 2.2]), respectively (p o 0.001). This seasonal trend was present all 3 years (Fig. 2). We observed an increased median number of sessions per day over the 3 years (2013-2014: 1500 sessions/d [IQR: 1115], 2014-2015: 1706 [IQR: 1334], 2015-2016: 1728 [IQR: 1352] (p o 0.001)). There was a corresponding increase in the overall number of sessions per year (20132014: 671,852 sessions, 2014-2015: 731,876 sessions, and 2015-2016: 731,493 sessions) (Fig. 3). The most popular time for access to the SCORE portal was between 6 PM and 6 AM: 38,011 (IQR: 4532) vs 17,529 (IQR: 19,850) during the day (p o 0.001), adjusted for local time zone. Peak use occurred between 8 and 10 PM (Fig. 4). Microsoft Windows was the most frequently used operating system at 48.9% (p o 0.001 vs others) (Fig. 5). About 70% of users accessed the portal from a desktop computer, 2013-2014
followed by tablet, and then mobile. Over the 3-year period, mobile sessions doubled from 7% to 13%, whereas tablet sessions decreased from 22% to 15% (Fig. 6).
DISCUSSION Several iterations of duty hour restrictions imposed by the ACGME have limited the amount of time that surgical trainees spend in the hospital, with an overall reduction of 6 to 12 months during the course of an individual resident’s training.8 Increasingly, asynchronous learning has become the model by which residents study for their clinical work and standardized examinations.9 It allows for flexibility and ease of use for trainees who are no longer allowed (or desire to remain) in the hospital in the evening. Additionally, many programs have adopted a night float system to 2014-2015
2015-2016
18000 16000 14000 12000 10000 8000 6000 4000 2000 0 1-Jul
1-Aug
1-Sep
1-Oct
1-Nov
1-Dec
1-Jan
1-Feb 1-Mar
1-Apr 1-May
1-Jun
FIGURE 3. Number of daily score sessions. Journal of Surgical Education Volume ]/Number ] ] 2017
3
2013-2014
2014-2015
2015-2016
60000 50000 40000 30000 20000 10000 0 0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23
Hour
FIGURE 4. Number of sessions per time of day.
accommodate, in particular, the 16-hour restriction on interns’ duty hours. Our data show that surgical residents do most of their SCORE studying in the evening with a peak between the hours of 8 and 10 PM, as adjusted for local time zone. Interestingly, there has been a recurring and precipitous decline in SCORE web portal usage after the ABSITE has been administered each January. This decline in use does not recover until the start of the new academic year. It is worth considering whether and how a change in the date of the ABSITE would affect SCORE web portal usage, and by proxy, the overall study habits of surgical residents. Other investigators have examined how residents study.10,11,12 Kim et al. reported that there was a divergence between how residents studied for their daily clinical duties vs preparation for the ABSITE. For the former, they were most likely to use a textbook (52%), the SCORE web portal (16%), or an ABSITE review book (15%). For the latter, they were most likely to use an ABSITE review book (48%), the SCORE portal question bank (17%), or the SCORE curriculum (14%). Our data indicate that residents’ SCORE portal usage (both in terms of number of sessions and session length) has steadily increased over this 3-year period, both before and after the time for
administration of the ABSITE. This was expressed as an increase in both the number of sessions and the duration of each session as the ABSITE approaches. We would postulate that several changes have encouraged the observed increase in use. First, starting in 2013, the ABS began mapping its general surgery exams (ABSITE, qualifying examination and certifying examination) to the SCORE general surgery curriculum outline. As of 2014, the topics in the outline, termed “modules” are, in turn, reported back to ABSITE examinees in their examination reports. Second, the establishment of TWIS in 2015 has provided ten peerreviewed, high-quality multiple choice questions for selfevaluation of the weekly material. Our analysis demonstrates a diurnal variation within each week, which, we believe, shows that many programs are building their didactic curriculum upon the TWIS schedule, which, in turn, is driving residents’ SCORE usage. Microsoft Windows was the most frequently used operating system at 48.9%, whereas Apple’s mobile and desktop platforms were each used by about a quarter of users. Until this year, the SCORE site has not been highly optimized for a mobile browser. However, it has recently been updated to provide a more user-friendly mobile view. Accordingly, we expect that mobile use will increase in the future. Device usage characteriscs by SCORE subscribers 80
2013-2014
2014-2015
2015-2016
400000 300000 200000 100000 0
71.5
72
69.5
70 60 50 40 30
21.5
20.1
20
10.4
7
10
14.8
13.2
0
2013 - 2014
2014 - 2015 Desktop
FIGURE 5. Operating system used. 4
Tablet
2015 - 2016 Mobile
FIGURE 6. Device usage characteristics by SCORE subscribers. Journal of Surgical Education Volume ]/Number ] ] 2017
There are some inherent limitations to this report. Included in this cohort of 42,743 subscribers were trainees from certain subspecialty programs (such as pediatric and vascular surgery), as well as some faculty. Although the vast majority of SCORE usage can be attributed to general surgery residents, it is important to acknowledge that other types of users were included in the dataset. We used Google Analytics to generate the user data for this study. Ad-filtering programs and extensions can block the tracking code used to populate the analytic software’s data. Additionally, the software uses a sampling methodology to limit reports to 500,000 randomly sampled sessions. This study was narrowly focused on surgical trainees’ SCORE usage. We suspect that, based on our data, residents are increasingly using SCORE to study for their day-to-day work and for ABSITE preparation. However, we have no reliable way of determining to what extent they rely on SCORE vs other media (textbooks, review books, and third-party online applications). There are several future investigations that may be fruitful in further delineating ideal study habits of surgical residents. We did not stratify SCORE portal use by individuals (this was aggregate data of all users over 3 years). It might be helpful to track a cohort of residents who have performed well on the ABSITE to elucidate a “model” template of ideal study habits. It is important to note that SCORE was not developed as an ABSITE preparation tool, but rather as a comprehensive curricular resource. The SCORE portal has developed into a ubiquitous learning tool among general surgery residency programs in the United States. Our analysis of user data demonstrates that general surgery residents are increasingly relying on it as a study tool throughout the year, and specifically for ABSITE preparation. Pending improvements to the SCORE portal—including mobile optimization—should help to encourage even more use. The editorial and content contributions from surgeons across the country continue to make SCORE an invaluable, home-grown resource to improve surgical education.
DISCLAIMER SCORE provided data to the authors for this study. However, this presentation does not necessarily reflect the opinions or policies of SCORE.
REFERENCES
2. Moalem J, Edhayan E, DaRosa DA, et al. Incorporat-
ing the SCORE curriculum and web site into your residency. J Surg Educ. 2011;68(4):294-297. 3. Jones AT, Biester TW, Buyske J, Lewis FR, Malan-
goni MA. Using the American Board of Surgery in training examination to predict board certification: a cautionary study. J Surg Educ. 2014;71(6):e144-e148. Epub 2014 Jun 7. 4. de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting
performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg. 2010;145(9):852-856. 5. de Virgilio C, Chan T, Kaji A, Miller K. Weekly
assigned reading and examinations during residency, ABSITE performance, and improved pass rates on the American Board of Surgery examinations. J Surg Educ. 2008;65(6):499-503. Available at: http://dx.doi.org/10. 1016/j.jsurg.2008.05.007. 6. Chang D, Kenel-Pierre S, Basa J, et al. Study habits
centered on completing review questions result in quantitatively higher American Board of Surgery in-training exam scores. J Surg Educ. 2014;71(6):e127-e131. Epub 2014 Aug 29. 7. Klingensmith ME, Jones AT, Smiley W, Biester TW,
Malangoni MA. Subscription to the Surgical Council on Resident Education web portal and qualifying examination performance. J Am Coll Surg. 2014;218 (4):566-570. 8. Lewis FR, Klingensmith ME. Issues in general surgery
residency training—2012. Ann Surg. 2012;256(4): 553-559. 9. Mallin M, Schlein S, Doctor S, Stroud S, Dawson M,
Fix M. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014;89(4): 598-601. 10. Derossis AM, DaRosa D, Schwartz A, Hauge LS,
Bordage G. Study habits of surgery residents and performance on American Board of Surgery in-training examinations. Am J Surg. 2004;188(3):230-236. 11. Yeh DD, Hwabejire JO, Imam A, et al. A survey of
study habits of general surgery residents. J Surg Educ. 2013;70(1):15-23. 12. Kim JJ, Kim DY, Kaji AH, et al. Reading habits of
organization devoted to graduate surgical education. J Am Coll Surg. 2007;204(3):341-346.
general surgery residents and association with American Board of Surgery in-training examination performance. J Am Med Assoc Surg. 2015;150(9):882-889.
Journal of Surgical Education Volume ]/Number ] ] 2017
5
1. Bell RH. Surgical council on resident education: a new
What Can SCORE Web Portal Usage Analytics Tell Us About How Surgical Residents Learn? Amit R.T. Joshi, MD, FACS,*,1 Aitua Salami, MD,* Mark Hickey, MA,†,2 Kerry B. Barrett, BA,†,2 Mary E. Klingensmith, MD, FACS,‡,1 and Mark A. Malangoni, MD, FACS§,1 Department of Surgery, Einstein Healthcare Network, Philadelphia, Pennsylvania; †Surgical Council on Resident Education, Philadelphia, Pennsylvania; ‡Department of Surgery, Washington University, St. Louis, Missouri; and §American Board of Surgery, Philadelphia, Pennsylvania *
PURPOSE: The Surgical Council on Resident Education
(SCORE) was established in 2004 with 2 goals: to develop a standardized, competency-based curriculum for general surgery residency training; and to develop a web portal to deliver this content. By 2012, 96% of general surgery residency programs subscribed to the SCORE web portal. Surgical educators have previously described the myriad ways they have incorporated SCORE into their curricula. The aim of this study was to analyze user data to describe how and when residents use SCORE. METHODS: Using analytic software, we measured SCORE usage from July, 2013 to June, 2016. Data such as IP addresses, geo-tagging coordinates, and operating system platforms were collected. The primary outcome was the median duration of SCORE use. Secondary outcomes were the time of day and the operating system used when accessing SCORE. Descriptive statistics were performed, and a p o 0.05 was deemed statistically significant.
(14.2) compared to July to October and March to June (12.3 [IQR: 3.2] and 9.6 minute/session [IQR: 2.2]), respectively (p o 0.001). This seasonal trend continued for all 3 years. We observed an increased number of sessions per day over the 3 years: median of 1500 sessions/d (IQR: 1115) vs 1706 (IQR: 1334) vs 1728 (IQR: 1352), p o 0.001. (Fig.). Most SCORE sessions occurred at night: 38,011 (IQR: 4532) vs 17,529 (IQR: 19,850) during the day (p o 0.001). Windows was the most frequently used operating system at 48.9% (p o 0.001 vs others). CONCLUSIONS: SCORE usage has increased significantly
over the last 3 years, when measured by number of sessions per day and length of time per session. There are predictable daily, diurnal, and seasonal variations in SCORE usage. The annual in-training examination is a prominent factor C 2017 stimulating SCORE usage. ( J Surg Ed ]:]]]-]]]. J Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
RESULTS: There were 42,743 total SCORE subscribers
KEY WORDS: surgery residents, study habits, ABSITE,
during the study period (75% resident and 25% faculty) with a mean of 14,248 subscribers per year. The overall median duration of SCORE use was 11.9 minute/session (interquartile range [IQR]: 6.8). Additionally, there was a significant increase in session length over the 3 academic years; 10.1 (IQR: 6.4), 11.9 (IQR: 7.2), and 13.2 minute/ session (IQR: 5.4) in 2013 to 2014, 2014 to 2015, and 2015 to 2016, respectively (p o 0.001). SCORE usage was highest in November to February at 21.0 minute/session
SCORE
This paper was presented as an oral abstract at the annual meeting of the APDS, San Diego, 2017. Correspondence: Inquiries to Amit R.T. Joshi, MD, FACS, Department of Surgery, Einstein Healthcare Network, 5401 Old York Road, Suite 510, Philadelphia, PA 19141; fax: þ(215)-663-6443; e-mail: [email protected] 1
Serve as consultants to SCORE.
2
Employees of SCORE.
COMPETENCIES: Medical Knowledge, Practice based Learning and Improvement, Professionalism
INTRODUCTION The Surgical Council on Resident Education (SCORE) was established in 2004 by a consortium of surgical organizations: the American Board of Surgery (ABS), American College of Surgeons, American Surgical Association, Association of Program Directors in Surgery, Association for Surgical Education, Accreditation Council for Graduate Medical Education (ACGME), and Society of American Gastrointestinal and Endoscopic Surgeons.1 SCORE developed a comprehensive, competency-based curriculum for general surgery residency training available through an
Journal of Surgical Education & 2017 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2017.10.002
1
online portal.2 Currently, 97% of the 272 ACGME-accredited general surgery residency training programs subscribe to SCORE. The SCORE web portal was launched in 2009 as a learning management system for general surgery residency programs and their trainees to use as a resource for study and education. The content in SCORE is organized around 750 discrete modules, which contain learning objectives, study questions, text and multimedia resources, and self-assessment questions. Since 2015, the SCORE web portal has provided an additional organizational framework whereby users and programs can cycle through the entire SCORE content in a defined 2-year curriculum. This sequence, This Week in SCORE (TWIS), includes a cluster of related modules paired with 10 peer-reviewed self-assessment questions to aid in preparation for each week’s educational content. Trainees and programs are able to track their performance on these weekly quizzes. TWIS is also used by many programs as the basis for didactic conferences. Investigators have previously studied correlations between in-training examinations and board certification.3-5 Chang et al.6 examined the sources used by residents. Prior work has demonstrated a considerable improvement in mean ABS qualifying examination scaled scores for residents in programs that initially subscribed to the SCORE web portal.7 The purpose of this report was to analyze SCORE portal usage data to determine how residents study on a daily and weekly basis and if patterns change during the academic year.
METHODS We used Google Analytics (Mountain View, CA) to measure SCORE portal use from July, 2013 to June, 2016. Once a
user has logged onto the SCORE portal, that user’s internet protocol (IP) address can generate an approximate geographic location. Data such as geo-tagging coordinates, session length, and operating system platforms were collected, in concordance with each user′s terms of service agreement that allows the collection of “aggregated, anonymous statistical information about use of the portal by licensees.” IP addresses were not linked with individual users. The primary outcome was the median duration of SCORE portal use as measured by logged-in time. Secondary outcomes were the time of day for logins and the operating system used for access. Descriptive statistics were performed, and a p o 0.05 was deemed statistically significant.
RESULTS There were 42,743 individual subscribers to the SCORE website during the 3-year study period. Subscribers comprised 75% resident and 25% faculty, with a mean of 14,248 subscribers per year from and a mean of 433 residency programs per year. IP address tagging indicated that subscribers overwhelmingly resided in the United States. There were, however, subscribers from every inhabited continent (Fig. 1). The overall median duration of SCORE use was 11.9 minute/session (interquartile range [IQR]: 6.8). Additionally, there was a significant increase in session duration over the 3 academic years; 10.1 (IQR: 6.4), 11.9 (IQR: 7.2), and 13.2 minute/session (IQR: 5.4) in 2013 to 2014, 2014 to 2015, and 2015 to 2016, respectively (p o 0.001). SCORE portal usage was highest from November to February at 21.0 minute/session (14.2) compared to July
FIGURE 1. Countries where the SCORE portal was accessed from 2013 to 2016. 2
Journal of Surgical Education Volume ]/Number ] ] 2017
2013-2014
2014-2015
2015-2016
60 50 40 30 20 10
Jul Jul Jul Aug Aug Sep Sep Sep Oct Oct Nov Nov Dec Dec Dec Jan Jan Feb Feb Mar Mar Mar Apr Apr May May Jun Jun Jun
0
FIGURE 2. Mean session length (min).
to October and March to June (12.3 [IQR: 3.2] and 9.6 minute/session [IQR: 2.2]), respectively (p o 0.001). This seasonal trend was present all 3 years (Fig. 2). We observed an increased median number of sessions per day over the 3 years (2013-2014: 1500 sessions/d [IQR: 1115], 2014-2015: 1706 [IQR: 1334], 2015-2016: 1728 [IQR: 1352] (p o 0.001)). There was a corresponding increase in the overall number of sessions per year (20132014: 671,852 sessions, 2014-2015: 731,876 sessions, and 2015-2016: 731,493 sessions) (Fig. 3). The most popular time for access to the SCORE portal was between 6 PM and 6 AM: 38,011 (IQR: 4532) vs 17,529 (IQR: 19,850) during the day (p o 0.001), adjusted for local time zone. Peak use occurred between 8 and 10 PM (Fig. 4). Microsoft Windows was the most frequently used operating system at 48.9% (p o 0.001 vs others) (Fig. 5). About 70% of users accessed the portal from a desktop computer, 2013-2014
followed by tablet, and then mobile. Over the 3-year period, mobile sessions doubled from 7% to 13%, whereas tablet sessions decreased from 22% to 15% (Fig. 6).
DISCUSSION Several iterations of duty hour restrictions imposed by the ACGME have limited the amount of time that surgical trainees spend in the hospital, with an overall reduction of 6 to 12 months during the course of an individual resident’s training.8 Increasingly, asynchronous learning has become the model by which residents study for their clinical work and standardized examinations.9 It allows for flexibility and ease of use for trainees who are no longer allowed (or desire to remain) in the hospital in the evening. Additionally, many programs have adopted a night float system to 2014-2015
2015-2016
18000 16000 14000 12000 10000 8000 6000 4000 2000 0 1-Jul
1-Aug
1-Sep
1-Oct
1-Nov
1-Dec
1-Jan
1-Feb 1-Mar
1-Apr 1-May
1-Jun
FIGURE 3. Number of daily score sessions. Journal of Surgical Education Volume ]/Number ] ] 2017
3
2013-2014
2014-2015
2015-2016
60000 50000 40000 30000 20000 10000 0 0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23
Hour
FIGURE 4. Number of sessions per time of day.
accommodate, in particular, the 16-hour restriction on interns’ duty hours. Our data show that surgical residents do most of their SCORE studying in the evening with a peak between the hours of 8 and 10 PM, as adjusted for local time zone. Interestingly, there has been a recurring and precipitous decline in SCORE web portal usage after the ABSITE has been administered each January. This decline in use does not recover until the start of the new academic year. It is worth considering whether and how a change in the date of the ABSITE would affect SCORE web portal usage, and by proxy, the overall study habits of surgical residents. Other investigators have examined how residents study.10,11,12 Kim et al. reported that there was a divergence between how residents studied for their daily clinical duties vs preparation for the ABSITE. For the former, they were most likely to use a textbook (52%), the SCORE web portal (16%), or an ABSITE review book (15%). For the latter, they were most likely to use an ABSITE review book (48%), the SCORE portal question bank (17%), or the SCORE curriculum (14%). Our data indicate that residents’ SCORE portal usage (both in terms of number of sessions and session length) has steadily increased over this 3-year period, both before and after the time for
administration of the ABSITE. This was expressed as an increase in both the number of sessions and the duration of each session as the ABSITE approaches. We would postulate that several changes have encouraged the observed increase in use. First, starting in 2013, the ABS began mapping its general surgery exams (ABSITE, qualifying examination and certifying examination) to the SCORE general surgery curriculum outline. As of 2014, the topics in the outline, termed “modules” are, in turn, reported back to ABSITE examinees in their examination reports. Second, the establishment of TWIS in 2015 has provided ten peerreviewed, high-quality multiple choice questions for selfevaluation of the weekly material. Our analysis demonstrates a diurnal variation within each week, which, we believe, shows that many programs are building their didactic curriculum upon the TWIS schedule, which, in turn, is driving residents’ SCORE usage. Microsoft Windows was the most frequently used operating system at 48.9%, whereas Apple’s mobile and desktop platforms were each used by about a quarter of users. Until this year, the SCORE site has not been highly optimized for a mobile browser. However, it has recently been updated to provide a more user-friendly mobile view. Accordingly, we expect that mobile use will increase in the future. Device usage characteriscs by SCORE subscribers 80
2013-2014
2014-2015
2015-2016
400000 300000 200000 100000 0
71.5
72
69.5
70 60 50 40 30
21.5
20.1
20
10.4
7
10
14.8
13.2
0
2013 - 2014
2014 - 2015 Desktop
FIGURE 5. Operating system used. 4
Tablet
2015 - 2016 Mobile
FIGURE 6. Device usage characteristics by SCORE subscribers. Journal of Surgical Education Volume ]/Number ] ] 2017
There are some inherent limitations to this report. Included in this cohort of 42,743 subscribers were trainees from certain subspecialty programs (such as pediatric and vascular surgery), as well as some faculty. Although the vast majority of SCORE usage can be attributed to general surgery residents, it is important to acknowledge that other types of users were included in the dataset. We used Google Analytics to generate the user data for this study. Ad-filtering programs and extensions can block the tracking code used to populate the analytic software’s data. Additionally, the software uses a sampling methodology to limit reports to 500,000 randomly sampled sessions. This study was narrowly focused on surgical trainees’ SCORE usage. We suspect that, based on our data, residents are increasingly using SCORE to study for their day-to-day work and for ABSITE preparation. However, we have no reliable way of determining to what extent they rely on SCORE vs other media (textbooks, review books, and third-party online applications). There are several future investigations that may be fruitful in further delineating ideal study habits of surgical residents. We did not stratify SCORE portal use by individuals (this was aggregate data of all users over 3 years). It might be helpful to track a cohort of residents who have performed well on the ABSITE to elucidate a “model” template of ideal study habits. It is important to note that SCORE was not developed as an ABSITE preparation tool, but rather as a comprehensive curricular resource. The SCORE portal has developed into a ubiquitous learning tool among general surgery residency programs in the United States. Our analysis of user data demonstrates that general surgery residents are increasingly relying on it as a study tool throughout the year, and specifically for ABSITE preparation. Pending improvements to the SCORE portal—including mobile optimization—should help to encourage even more use. The editorial and content contributions from surgeons across the country continue to make SCORE an invaluable, home-grown resource to improve surgical education.
DISCLAIMER SCORE provided data to the authors for this study. However, this presentation does not necessarily reflect the opinions or policies of SCORE.
REFERENCES
2. Moalem J, Edhayan E, DaRosa DA, et al. Incorporat-
ing the SCORE curriculum and web site into your residency. J Surg Educ. 2011;68(4):294-297. 3. Jones AT, Biester TW, Buyske J, Lewis FR, Malan-
goni MA. Using the American Board of Surgery in training examination to predict board certification: a cautionary study. J Surg Educ. 2014;71(6):e144-e148. Epub 2014 Jun 7. 4. de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting
performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg. 2010;145(9):852-856. 5. de Virgilio C, Chan T, Kaji A, Miller K. Weekly
assigned reading and examinations during residency, ABSITE performance, and improved pass rates on the American Board of Surgery examinations. J Surg Educ. 2008;65(6):499-503. Available at: http://dx.doi.org/10. 1016/j.jsurg.2008.05.007. 6. Chang D, Kenel-Pierre S, Basa J, et al. Study habits
centered on completing review questions result in quantitatively higher American Board of Surgery in-training exam scores. J Surg Educ. 2014;71(6):e127-e131. Epub 2014 Aug 29. 7. Klingensmith ME, Jones AT, Smiley W, Biester TW,
Malangoni MA. Subscription to the Surgical Council on Resident Education web portal and qualifying examination performance. J Am Coll Surg. 2014;218 (4):566-570. 8. Lewis FR, Klingensmith ME. Issues in general surgery
residency training—2012. Ann Surg. 2012;256(4): 553-559. 9. Mallin M, Schlein S, Doctor S, Stroud S, Dawson M,
Fix M. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014;89(4): 598-601. 10. Derossis AM, DaRosa D, Schwartz A, Hauge LS,
Bordage G. Study habits of surgery residents and performance on American Board of Surgery in-training examinations. Am J Surg. 2004;188(3):230-236. 11. Yeh DD, Hwabejire JO, Imam A, et al. A survey of
study habits of general surgery residents. J Surg Educ. 2013;70(1):15-23. 12. Kim JJ, Kim DY, Kaji AH, et al. Reading habits of
organization devoted to graduate surgical education. J Am Coll Surg. 2007;204(3):341-346.
general surgery residents and association with American Board of Surgery in-training examination performance. J Am Med Assoc Surg. 2015;150(9):882-889.
Journal of Surgical Education Volume ]/Number ] ] 2017
5
1. Bell RH. Surgical council on resident education: a new