- Email: [email protected]
The Social Media Index as an Indicator of Quality for Emergency Medicine Blogs: A METRIQ Study
EDUCATION/BRIEF RESEARCH REPORT
The Social Media Index as an Indicator of Quality for Emergency Medicine Blogs: A METRIQ Study Brent Thoma, MD, MA*; Teresa M. Chan, MD, MHPE; Puneet Kapur, MD MSc; Derek Sifford; Marshall Siemens; Michael Paddock, DO; Felix Ankel, MD; Andy Grock, MD; Michelle Lin, MD; for the METRIQ Study Collaborators* *Corresponding Author. E-mail: [email protected], Twitter: @Brent_Thoma.
Study objective: Online educational resources such as blogs are increasingly used for education by emergency medicine clinicians. The Social Media Index was developed to quantify their relative impact. The Medical Education Translational Resources: Indicators of Quality (METRIQ) study was conducted in part to determine the association between the Social Media Index score and quality as measured by gestalt and previously derived quality instruments. Methods: Ten blogs were randomly selected from a list of emergency medicine and critical care Web sites. The 2 most recent clinically oriented blog posts published on these blogs were evaluated with gestalt, the Academic Life in Emergency Medicine Approved Instructional Resources (ALiEM AIR) score, and the METRIQ-8 score. Volunteer raters (including medical students, emergency medicine residents, and emergency medicine attending physicians) were identified with a multimodal recruitment methodology. The Social Media Index was calculated in February 2016, November 2016, April 2017, and December 2017. Pearson’s correlations were calculated between the Social Media Index and the average rater gestalt, ALiEM AIR score, and METRIQ-8 score. Results: A total of 309 of 330 raters completed all ratings (93.6%). The Social Media Index correlated moderately to strongly with the mean rater gestalt ratings (range 0.69 to 0.76) and moderately with the mean rater ALiEM AIR score (range 0.55 to 0.61) and METRIQ-8 score (range 0.53 to 0.57) during the month of the blog post’s selection and for 2 years after. Conclusion: The Social Media Index’s correlation with multiple quality evaluation instruments over time supports the hypothesis that it is associated with overall Web site quality. It can play a role in guiding individuals to high-quality resources that can be reviewed with critical appraisal techniques. [Ann Emerg Med. 2018;-:1-7.] Please see page XX for the Editor’s Capsule Summary of this article. 0196-0644/$-see front matter Copyright © 2018 by the American College of Emergency Physicians. https://doi.org/10.1016/j.annemergmed.2018.05.003
INTRODUCTION Background The increasing number of educational Web sites focused on emergency medicine1 corresponds to the increased use of these resources by medical trainees.2 Despite their popularity, there is skepticism in regard to their quality.3 Unfortunately, individual gestalt has been shown to be unreliable in evaluating their quality with fewer than 42 gestalt raters.4 Furthermore, recommendations that faculty guide trainees in resource selection2 are incompatible with the ad hoc manner in which these resources are accessed, and findings suggest that attending physician gestalt is no more reliable than that of their trainees.4 Blog post quality assessment instruments such as the Academic Life in Emergency Medicine Approved Instructional Resources (ALiEM AIR) (Appendix E1, *All members are listed in the Appendix.
Volume
-,
no.
-
:
-
2018
available online at http://www.annemergmed.com)5 and Medical Education Translational Resources: Impact and Quality (METRIQ-8) instruments (Appendix E2, available online at http://www.annemergmed.com)6 aim to guide the appraisal of these resources much like a critical appraisal guide for medical literature. However, there is little evidence supporting their use by general users. Validity evidence for the use of ALiEM AIR consists of its relatively reliable use by a small population of experienced medical education experts,5 whereas METRIQ-8 is supported through its evidence-based validation.6 Both instruments have demonstrated reliability similar to that of gestalt.7 The Social Media Index was developed to quantify the relative impact of blogs and podcasts by amalgamating measures of followership, which include Twitter follows, Facebook likes, and the Alexa score (a measure of Web site readership).8 It is a Web site–level, normalized, logarithmic, 10-point scale that is published for greater Annals of Emergency Medicine 1
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs
Editor’s Capsule Summary
What is already known on this topic Assessing the quality of medical education blogs presents challenges. Gestalt assessments and structured scoring instruments have demonstrated some validity. What question this study addressed The authors compared 4 measurements of the Social Media Index, which ranks blogs according to their overall popularity online, with ratings of 20 blog posts by 309 raters using gestalt assessments and 2 scoring instruments. What this study adds to our knowledge In assessing the abstract and undefined educational value of blogs, the authors found a fair correlation between the Social Media Index and scoring systems, and a somewhat stronger correlation with gestalt recommendations. How this is relevant to clinical practice Although no measures perfectly identify quality medical education blogs, using the Social Media Index to measure overall popularity correlates moderately well with other assessments of individual blog post quality.
than 150 Web sites twice per year on the ALiEM Web site (Figure 1). The Social Media Index aims to provide trainees with a list of prominent blogs and podcasts, educators and institutions with a relative measure of impact, and researchers with a way to stratify these resources.8 When applied to medical journals, the Social Media Index correlates well with traditional measures of impact.8 The Social Media Index has been criticized because its score may be more related to popularity than quality.9 We sought to determine whether it could approximate the quality of blog posts by investigating its correlation with the mean gestalt, ALiEM AIR, and METRIQ-8 scores provided by a general population of medical raters. MATERIALS AND METHODS The METRIQ study is a cross-sectional rating study that evaluated 20 blog posts from 10 blogs, using 4 instruments: the Social Media Index score, gestalt rating, the ALiEM AIR score, and the METRIQ-8 score. Multiple analyses and publications, including this study, were planned a priori with the same data set including the 2 Annals of Emergency Medicine
recruitment methodology,10 reliability of the gestalt data,4 and reliability of the METRIQ-8 and ALiEM AIR scores.7 The METRIQ study protocol met the University of Saskatchewan’s Research Ethics Board requirements for exemption (BEH 16-09). On February 24, 2016, 10 blogs (Appendix E3, available online at http://www.annemergmed.com) were selected with a random-number generator (https://www. randomizer.org/) from a comprehensive list of emergency medicine and critical care Web sites that was created1 in 2013 and updated with a Google search in February 2016. In this update, one author (M.S.) reviewed the first 500 results for “emergency medicine” AND “blog” OR “podcast” and added Web sites written in English that had published a clinical blog post within the past 6 months. Web sites that had not published clinical blog posts since January 1, 2016, were excluded, along with blog posts consisting only of the “show-notes” of an accompanying podcast. In this review, one author (M.S.) identified Web sites for exclusion and another (B.T.) reviewed them. To avoid selection bias, each Web site’s 2 most recently published clinical blog posts were used. As described below and elsewhere in greater detail,10 a convenience sample of medical students, emergency medicine residents, and emergency medicine attending physicians was recruited through a multimodal recruitment strategy. A power calculation was initially conducted to guide enrollment for the gestalt analyses.4 However, because of the multiple complex analyses planned, the focus was on recruiting the maximum number of participants and incorporating the maximum feasible number of blog posts rather than a specific target. The recruitment strategy included in-person and e-mail communications with potential raters, as well as a social media strategy using Facebook and Twitter, a YouTube video, blog posts on the METRIQ Study Web site, and a podcast and blog hosted on the Skeptic’s Guide to Emergency Medicine. Raters who rated all blog posts are recognized as METRIQ study collaborators. Recruitment occurred between March 1 and April 30, 2016. The rating period was open between March 1 and June 1, 2016. Potential raters were directed to the METRIQ Study Web site (http://metriqstudy.org/) to complete a study intake form. A random-number generator (https://www.randomizer.org/) was used to generate blocks of numbers from 1 to 4 with no repeated values. Medical students, residents, and attending physicians were assigned to survey versions consecutively within separate blocks. Within 24 hours, these potential raters were e-mailed the information needed to consent to and enroll in the study. From that point until they completed rating, they were sent Volume
-,
no.
-
:
-
2018
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs
Figure 1. SMi formula for a given Web site. SMi, Social Media Index.
a reminder e-mail every 7 to 10 days a maximum of 4 times. In addition, a final notification was sent to all raters that the study was closing within 10 days. The online rating survey (Appendix E4, available online at http://www.annemergmed.com) was created with FluidSurveys.com. Four authors of the METRIQ study (including B.T., T.C., and M.L.) conducted an internal pilot of the survey that resulted in minor changes and estimated its length to be 90 minutes. This was thought to be too long for a single session, so the option to start and stop the survey at the raters’ convenience was added. The survey asked raters to rate each of the 20 blog posts for quality on a 7-point Likert scale, using gestalt. These data were collected alongside blog post evaluations with one or both of the ALiEM AIR and METRIQ-8 instruments.5,6 Each rater rated 10 blog posts with both instruments, 5 with only ALiEM AIR and 5 with only METRIQ-8. To minimize the effect that the ratings would have on gestalt evaluation, raters were instructed to provide their gestalt rating before using the instruments. Four versions of the survey were developed to minimize order effect and ensure that each blog post was rated approximately the same number of times with the ALiEM AIR and METRIQ-8 instruments. The Social Media Index score was calculated for the entire list of Web sites in February 2016, November 2016, April 2017, and December 2017,8 using the updated formula published on the ALiEM Web site (https://aliem. com/social-media-index). This version of the formula was modified after the previous publication8 because Google PageRank is no longer supported. Primary Data Analysis Descriptive statistics were calculated for the rater demographics and the ratings of each blog post. Raters who did not rate all assigned blog posts were excluded. Pearson’s correlations were calculated for the blog posts between the Social Media Index score (February 2016, November 2016, June 2017, and December 2017) and the average rater gestalt, ALiEM AIR, and METRIQ-8 scores of the raters. Volume
-,
no.
-
:
-
2018
Modified Bland-Altman plots were created to visually compare the mean Social Media Index score with mean rater gestalt rating, mean ALiEM AIR score, and mean METRIQ-8 score. For this comparison, each of the comparison variables was normalized on a 10-point scale. These figures are not traditional Bland-Altman plots because they compare aggregate variables and use mean values rather than individual data points. RESULTS After the Google search and application of the exclusion criteria, 60 Web sites were eligible for the study and 10 were selected. A total of 309 of the 330 potential raters who consented to the study completed all ratings (93.6%). Each blog post was rated with gestalt by 309 raters and using the ALiEM AIR and METRIQ-8 instruments by 229 to 236 raters. Raters were recruited from 27 countries, with the preponderance from North America. Most raters (75.0%) read blog posts for medical education purposes at least once per month. The raters’ demographics are presented in Table 1. With Pearson’s correlation calculations, the Social Media Index score correlated moderately to strongly with mean gestalt ratings (range 0.69 to 0.76) and moderately with mean ALiEM AIR scores (range 0.55 to 0.61) and METRIQ-8 scores (range 0.53 to 0.57). These correlations persisted for nearly 2 years after the month of the blog post’s publication (Table 2). Modified Bland-Altman plots visually representing the relationship between the mean Social Media Index with the normalized mean gestalt, ALiEM AIR score, and METRIQ-8 score are presented in Figure 2. The y axis displays the difference between the Social Media Index score and the normalized score for each blog, whereas the x axis displays the average of the Social Media Index score and the normalized score for each blog post. The negative average value for each score on the y axis indicates that the Social Media Index generally underestimates a blog post’s quality relative to the other metrics. This effect is more pronounced for lower-quality blog posts. Annals of Emergency Medicine 3
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs Table 1. Rater demographics and characteristics (n ¼ 309). Variable
No. (%)
Level of training Medical students
121 (39.2)
Emergency medicine residents
88 (28.5)
Emergency medicine attending physicians
100 (32.4)
Age, mean (SD), y
31.1 (7.3)
Sex Male
184 (59.5)
Female
123 (39.8)
Other
2 (0.6)
How often do you read blog posts on medical topics for educational purposes? Every day
48 (15.5)
Several times a week
141 (45.6)
Once a week
43 (13.9)
Several times a month
38 (12.3)
Once a month
21 (6.8)
Less than once a month
15 (4.9)
Never
2 (0.6)
No response
1 (0.3)
Do you manage, edit, own, or operate a blog that posts on medical topics for educational purposes? Yes
45 (14.5)
No
261 (84.5)
No response
3 (0.6)
Country Canada
146 (47.2)
United States
111 (35.9)
Other (25 countries)
49 (15.9)
LIMITATIONS First, there is no universally recognized definition of quality with which the Social Media Index may be compared. We addressed this by including the reference standard of the average gestalt of many raters,4 along with the only 2 quality evaluation instruments that have been derived to appraise blog posts targeting clinicians.5,6 Lacking true values for quality, these scores provide multiple quantitative perspectives to provide as comprehensive an evaluation of the
Social Media Index as possible. Second, previous studies have demonstrated that gestalt has limited application because large numbers of raters must be used to obtain reliable values.4 However, enough raters were included in this study to ensure an accurate comparison. Third, raters were not blinded to the Web site domain for the featured blog posts. This may have resulted in their rating familiar or high-profile blogs more favorably than others. Fourth, the dates that the Social Media Index score was calculated were not standardized. Because this would be more likely to increase the variability in Social Media Index calculations because of annual variations in viewership, it is unlikely to have led to spurious correlations. Fifth, only 2 blog posts from a single month were evaluated, which may not have been representative of all blog posts on a Web site and could have skewed the quality ratings of some Web sites, affecting their correlation with the Social Media Index in an undetermined direction. DISCUSSION Quality assessment of online resources is challenging, especially for individuals and learners. High number of raters are required to generate reliable gestalt ratings4 and the AIR score has been studied only in a group of trained attending physicians.5 The METRIQ study was conducted to test our hypothesis that Web sites identified by the Social Media Index as being more influential would also be of higher quality.8 Our results demonstrated a consistent correlation between the Social Media Index score and the average ratings of blog posts from a large group of raters using gestalt and 2 quality evaluation instruments longitudinally over time. Because a criterion standard for quality in medical education blogs is lacking, quality was assessed from the perspectives of all 3 of the approaches published in the literature. As the ratings used in this data set have demonstrated reliability, they can be assumed to reflect an accurate quality assessment.4,7 These findings suggest that the Social Media Index score can be an important surrogate marker of quality that can guide individuals and small groups who would otherwise be unable to reliably assess resource quality.
Table 2. Pearson’s correlations between the Social Media Index score, gestalt, ALiEM AIR score, and METRIQ-8 score. SMi (95% CI) February 2016
November 2016
April 2017
December 2017
Range
Gestalt
0.69 (0.34–1.00)
0.72 (0.38–1.00)
0.70 (0.35–1.00)
0.76 (0.44–1.00)
0.69–0.76
ALiEM AIR
0.55 (0.13–0.96)
0.56 (0.15–0.97)
0.55 (0.13–0.96)
0.61 (0.21–1.00)
0.55–0.61
METRIQ-8
0.53 (0.12–0.95)
0.56 (0.15–0.97)
0.52 (0.10–0.95)
0.57 (0.16–0.97)
0.53–0.57
CI, confidence interval.
4 Annals of Emergency Medicine
Volume
-,
no.
-
:
-
2018
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs
does correlate with multiple interpretations of quality and that the relationship between quality and the Social Media Index score is persistent. The correlation between the Social Media Index and quality assessments makes sense. Web sites with lowerquality content may be less likely to be viewed, shared, and “liked” by clinicians seeking reliable resources. We suspect that clinicians are also less likely to consistently visit Web sites that they think are of lesser quality. Thus, the Social Media Index score may serve as a crowdsourced gestalt assessment of the quality of a Web site’s longitudinal content. However, Figure 2 demonstrates that, relative to the quality evaluations, the Social Media Index underestimates the quality of lower-scoring resources. This suggests that there may be high-quality resources that are not identified by the Social Media Index because they are less influential. Consideration was given to correlating subgroups of raters (medical students, emergency medicine residents, and emergency medicine attending physicians) with the Social Media Index separately. However, because the average ratings of these groups was so strongly correlated (Pearson’s correlation >0.92 between the subgroups for all 3 ratings),4 the subgroup results were not expected to differ from those of the larger group. The Social Media Index score for educational Web sites in emergency medicine and critical care was shown to correlate reasonably well with quality evaluations of blog content. Although critical appraisal of article-level online content is still warranted, this study supports the use of the Social Media Index as a tool to guide individuals and groups toward high-quality social media Web sites. Supervising editor: N. Seth Trueger, MD, MPH
Figure 2. Modified Bland-Altman plots comparing the mean SMi value with the normalized mean gestalt rating, mean ALiEM AIR score, and mean METRIQ-8 score for each blog post.
Although the Social Media Index score correlated reasonably well with gestalt, ALiEM AIR, and METRIQ-8, its correlation with gestalt was strongest, possibly because certain aspects of raters’ preferences are more aptly captured by gestalt (eg, design, layout) than criterion-based scores. Moreover, there may have been an inevitable halo effect when raters evaluated blog posts on more “popular” or influential Web sites as having higher quality because of greater familiarity with these educational resources. A consistent positive correlation was found between the Social Media Index score and both criterion-based tools over time. This suggests that the Social Media Index score Volume
-,
no.
-
:
-
2018
Author affiliations: From the Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada (Thoma, Kapur, Siemens); Health Professions Education, Massachusetts General Hospital Institute, Boston, MA (Thoma); the Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, Ontario, Canada (Chan); Wayne State University, Detroit, MI (Sifford); the Department of Emergency Medicine, Regions Hospital, HealthPartners Institute, Saint Paul, MN, and the Department of Emergency Medicine, University of Minnesota, Saint Paul, MN (Paddock, Ankel); the HealthPartners Institute, Saint Paul, MN (Ankel); the USCþLAC Department of Emergency Medicine and David Geffen School of Medicine at UCLA, Los Angeles, CA (Grock); and the Department of Emergency Medicine, University of California–San Francisco, San Francisco, CA (Lin). Author contributions: BT, TC, PK, DS, MS, and ML conceived the study and designed the trial. BT obtained research funding, supervised the conduct of the trial, and drafted the article. BT, PK, Annals of Emergency Medicine 5
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs and MS performed data collection. BT, TC, MS, MP, FA, AG, and ML recruited raters. TC provided statistical advice on study design and analysis. BT and TC analyzed the data. All authors contributed substantially to article revision. BT takes responsibility for the paper as a whole. All authors attest to meeting the four ICMJE.org authorship criteria: (1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND (2) Drafting the work or revising it critically for important intellectual content; AND (3) Final approval of the version to be published; AND (4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org). Funding for this research was provided by the Canadian Association of Emergency Physicians (Junior Investigator Grant) and the Royal College of Physicians and Surgeons of Canada (Robert Maudsley Fellowship for Studies in Medical Education). The following authors report operating medical education blogs: Brent Thoma (ALiEM.com, CanadiEM.org, Debrief2Learn.org), Teresa Chan (ALiEM.com and CanadiEM.com), Derek Sifford (ALiEM.com), and Michelle Lin (ALiEM.com). Publication dates: Received for publication January 9, 2018. Revisions received February 9, 2018, and March 31, 2018. Accepted for publication May 1, 2018.
REFERENCES 1. Cadogan M, Thoma B, Chan TM, et al. Free open access meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002-2013). Emerg Med J. 2014;31:e76-e77. 2. Mallin M, Schlein S, Doctor S, et al. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014;89:598-601. 3. Cameron P. Pundit-based medicine. Available at: http://www. epijournal.com/articles/240/pundit-based-medicine. Accessed January 15, 2016. 4. Thoma B, Sebok-Syer SS, Krishnan K, et al. Individual gestalt is unreliable for the evaluation of quality in medical education blogs: a METRIQ study. Ann Emerg Med. 2017;70:394-401. 5. Chan TM, Grock A, Paddock M, et al. Examining reliability and validity of an online score (ALiEM AIR) for rating free open access medical education resources. Ann Emerg Med. 2016;68:729-735. 6. Chan T, Thoma B, Krishnan K, et al. The derivation of two simplified critical appraisal scores for use by trainees to evaluate online educational resources: a METRIQ study. West J Emerg Med. 2016;XVII:574-584. 7. Sebok-Syer SS. Colmers-Gray I, et al. Quality evaluation scores are no more reliable than gestalt in evaluating the quality of emergency medicine blogs: a METRIQ study. Teach Learn Med. 2018;30:294-302. 8. Thoma B, Sanders JL, Lin M, et al. The Social Media Index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015;16:242-249. 9. Carley S. The Social Media Index (SMi): can and should we measure FOAMed? St Emlyn’s Blog. Available at: http://stemlynsblog.org/ the-social-media-index-smi-is-it-flawed/. Accessed April 11, 2016.
6 Annals of Emergency Medicine
10. Thoma B, Paddock M, Purdy E, et al. Leveraging a virtual community of practice to participate in a survey-based study: a description of the METRIQ study methodology. AEM Educ Train. 2017;1:110-113.
APPENDIX The METRIQ Study Collaborators (in alphabetical order) Charlotte Alexander, Mohammed Alkhalifah, Abdulaziz S. Almehlisi, Saeed Alqahtani, Scott Anderson, Shelaina Anderson, Colin Andrews, Jocelyn Andruko, Felix Ankel, Nikytha Antony, Diptesh Aryal, Barbra Backus, Jennifer Baird, Andrew Baker, Sarah Batty, Jared Baylis, Braeden Beaumont, Chris Belcher, Brent Benavides, Michael Benham, Julian Botta, Nicholas Bouchard, Victoria Brazil, Emily Brumfield, Anthony Bryson, Wisarut Bunchit, Kat Butler, Lindy Buzikievich, David Calcara, Rob Carey, Stephen Carroll, Louise Cassidy, Kirsty Challen, Kathryn Chan, Tim Chaplin, Natasha Chatham-Zvelebil, Eric Chen, Lucy Chen, Sushant Chhabra, Alvin Chin, Eric Chochi, Tina Choudhri, Jeremy Christensen, Isabelle Colmers-Gray, Kimberly Connors, Veronica Coppersmith, Abby Cosgrove, Gregory Costello, Kevin Cullison, Andrew D’Alessandro, Kerstin de Wit, Marie Decock, Rayan Delbani, William Denq, Julianna Deutscher, Brendan Devine, Maia Dorsett, Taylor Duda, Justin Dueweke, Teresa Dunphy, Sean Dyer, Karthryn T Eastley, Marcia Edmonds, Ken Edwards, Robert Ehrman, Youness Elkhalidy, Preston Fedor, Brian Ficiur, Caley Flynn, Bill Fraser, Meagan Fu, James Fukakusa, Eric Funk, Damjan Gaco, Viktor Gawlik, Kenn Ghaffarian, Laleh Gharahbaghian, Phil Griffith, Andrew Griffith, Andrew Grock, Tanner Gronowski, Cathy Grossman, Jaroslaw Gucwa, Pawan Gupta, Alexandra Gustafson, Andrew Guy, Mary Haas, Stanislaw Haciski, Emina Hajdinjak, Andrew K. Hall, Regina Hammock, Jan Hansel, Alexander Hart, Larissa Hattin, Brandon Herb, SueLin Hilbert, Jesse Hill, Jeff Hill, Amy Ho, Emily House, Nina House, James Huffman, Charlie Inboriboon, Alex Ireland, Ali Jamal, Mohammad Ali Jamil, Victor Jansen, Zach Jarou, Vivian Jia, Levi Johnston, Drew Kalnow, Puneet Kapur, Seth Kelly, Kyle Kelson, William Kent, Rishi Khakhkhar, Jaasmit Khurana, Ashley Kilp, Scott Knapp, Sebastian Kohler, Ivanna Kruhlak, Nadim Lalani, Samantha Lam, Patrick Lank, Zander Laurie, Kristina Lea, Ernest Leber, Ching-Hsing Lee, Haakon Lenes, Nilantha Lenora, Jesse Leontowicz, Kelly Lien, Yingchun Lin, Michelle Lin, Andrew Little, Ivy Liu, Harry Liu, Steve Liu, Stephanie Louka, Elise Lovell, David Lowe, Ashley Lubberdink, Jessica Luc, Casey Lyons, Sheng-Hsiang Ma, Hugh MacLeod, Nick Mancuso, Anali Maneshi, Jesse May, John Mayo, Mike McDonnell, Susan McLellan, Carolyn Volume
-,
no.
-
:
-
2018
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs
McQuarrie, Julia Nood, Therese Mead, Cory Meeuwisse, Patrick Meloy, Perry Menzies, Anne Messman, Stephen Miazga, Logan Mills, Ken Milne, Allan Mix, Steve Montag, Brendan Moore, Justin Morgenstern, Sarah Mott, P. Mukherj, Ali Mulla, Sheena Nandalal, Taylor Nikel, Sean Nugent, Morgan Oakland, Werner Oberholzer, Onyeka Otugo, Taofiq Segun Oyedokun, Mike Paddock, Alim Pardhan, Kinjal Patel, Quinten Paterson, Catherine Patocka, Christine Patterson, James Pearlman, Elyse Berger Pelletier, Alexis Pelletier-Bui, Marc Phan, Zafrina Poonja, Aubrey Powell, Kamini Premkumar, Gregor Prosen, Vishal Puri, Tanis Quaife, Ryan Raffel, Ali Raja, Randi Ramunno, Louise Rang, Suzanne Rannazzisi, Shauna Regan, Salim R. Rezaie, Milan Ridderikhof, Vanessa Rogers, Christine Roh, Dra. Maria Rosa Carrillo, Keith Rosenberg, Marina Roure, Sherri Rudinsky, Joshua Rudner, Adeeb Saleh, Will Sanderson, Owen Scheirer, Paul Schofield, Paul Schunk, Evan Schwarz, Parisa Shahrabadi, Eric Shappell, Julia Sheffield, Jonathan Sherbino, Manpreet Singh, Hector C
Volume
-,
no.
-
:
-
2018
Singson, Dave Slessor, Sam Smith, Paula Sneath, Robert Sobehart, Kerry Spearing, James Stempien, Britni Sternard, Tara Stratton, Katherine Stuart, Bob Stuntz, Michael Susalla, Colleen Sweeney, Loice Swisher, Henry Swoboda, Shahbaz Syed, Taku Taira, Nikhil Tambe, Richard Tang, Elisha Targonsky, Rachel Taylor, Alan Taylor, Todd Taylor, Paxton Ting, Gerhard Tiwald, Kelvin Tran, Evelyn Tran, Jason Trickovic, Paul Trinquero, Seth Trueger, Aaron Tyagi, Manrique Umana, Patrick Vallance, Patricia Van den Berg, Luis Vargas, Rene Verbeek, Sandra Viggers, Zlata Vlodaver, Matthew Wagner, Noorin Walji, Joe Walter, Miranda Wan, Rachel Wang, Gregory Wanner, Wyatt Warawa, Mike Ward, Jennifer Weekes, Kristen Weersink, Cara Weessies, Anna Whalen-Browne, Brian Whiteside, Matthew Willis, Jonas Wilmer, Nelson Wong, Mark Woodcroft, Rob Woods, Lawrence Yau, Jessica Yee, Calvin Yeh, Simon York Ming Huang, Katherine Yurkiw, Fareen Zaver, and Alexander Zozula
Annals of Emergency Medicine 7
The Social Media Index as an Indicator of Quality for Emergency Medicine Blogs: A METRIQ Study Brent Thoma, MD, MA*; Teresa M. Chan, MD, MHPE; Puneet Kapur, MD MSc; Derek Sifford; Marshall Siemens; Michael Paddock, DO; Felix Ankel, MD; Andy Grock, MD; Michelle Lin, MD; for the METRIQ Study Collaborators* *Corresponding Author. E-mail: [email protected], Twitter: @Brent_Thoma.
Study objective: Online educational resources such as blogs are increasingly used for education by emergency medicine clinicians. The Social Media Index was developed to quantify their relative impact. The Medical Education Translational Resources: Indicators of Quality (METRIQ) study was conducted in part to determine the association between the Social Media Index score and quality as measured by gestalt and previously derived quality instruments. Methods: Ten blogs were randomly selected from a list of emergency medicine and critical care Web sites. The 2 most recent clinically oriented blog posts published on these blogs were evaluated with gestalt, the Academic Life in Emergency Medicine Approved Instructional Resources (ALiEM AIR) score, and the METRIQ-8 score. Volunteer raters (including medical students, emergency medicine residents, and emergency medicine attending physicians) were identified with a multimodal recruitment methodology. The Social Media Index was calculated in February 2016, November 2016, April 2017, and December 2017. Pearson’s correlations were calculated between the Social Media Index and the average rater gestalt, ALiEM AIR score, and METRIQ-8 score. Results: A total of 309 of 330 raters completed all ratings (93.6%). The Social Media Index correlated moderately to strongly with the mean rater gestalt ratings (range 0.69 to 0.76) and moderately with the mean rater ALiEM AIR score (range 0.55 to 0.61) and METRIQ-8 score (range 0.53 to 0.57) during the month of the blog post’s selection and for 2 years after. Conclusion: The Social Media Index’s correlation with multiple quality evaluation instruments over time supports the hypothesis that it is associated with overall Web site quality. It can play a role in guiding individuals to high-quality resources that can be reviewed with critical appraisal techniques. [Ann Emerg Med. 2018;-:1-7.] Please see page XX for the Editor’s Capsule Summary of this article. 0196-0644/$-see front matter Copyright © 2018 by the American College of Emergency Physicians. https://doi.org/10.1016/j.annemergmed.2018.05.003
INTRODUCTION Background The increasing number of educational Web sites focused on emergency medicine1 corresponds to the increased use of these resources by medical trainees.2 Despite their popularity, there is skepticism in regard to their quality.3 Unfortunately, individual gestalt has been shown to be unreliable in evaluating their quality with fewer than 42 gestalt raters.4 Furthermore, recommendations that faculty guide trainees in resource selection2 are incompatible with the ad hoc manner in which these resources are accessed, and findings suggest that attending physician gestalt is no more reliable than that of their trainees.4 Blog post quality assessment instruments such as the Academic Life in Emergency Medicine Approved Instructional Resources (ALiEM AIR) (Appendix E1, *All members are listed in the Appendix.
Volume
-,
no.
-
:
-
2018
available online at http://www.annemergmed.com)5 and Medical Education Translational Resources: Impact and Quality (METRIQ-8) instruments (Appendix E2, available online at http://www.annemergmed.com)6 aim to guide the appraisal of these resources much like a critical appraisal guide for medical literature. However, there is little evidence supporting their use by general users. Validity evidence for the use of ALiEM AIR consists of its relatively reliable use by a small population of experienced medical education experts,5 whereas METRIQ-8 is supported through its evidence-based validation.6 Both instruments have demonstrated reliability similar to that of gestalt.7 The Social Media Index was developed to quantify the relative impact of blogs and podcasts by amalgamating measures of followership, which include Twitter follows, Facebook likes, and the Alexa score (a measure of Web site readership).8 It is a Web site–level, normalized, logarithmic, 10-point scale that is published for greater Annals of Emergency Medicine 1
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs
Editor’s Capsule Summary
What is already known on this topic Assessing the quality of medical education blogs presents challenges. Gestalt assessments and structured scoring instruments have demonstrated some validity. What question this study addressed The authors compared 4 measurements of the Social Media Index, which ranks blogs according to their overall popularity online, with ratings of 20 blog posts by 309 raters using gestalt assessments and 2 scoring instruments. What this study adds to our knowledge In assessing the abstract and undefined educational value of blogs, the authors found a fair correlation between the Social Media Index and scoring systems, and a somewhat stronger correlation with gestalt recommendations. How this is relevant to clinical practice Although no measures perfectly identify quality medical education blogs, using the Social Media Index to measure overall popularity correlates moderately well with other assessments of individual blog post quality.
than 150 Web sites twice per year on the ALiEM Web site (Figure 1). The Social Media Index aims to provide trainees with a list of prominent blogs and podcasts, educators and institutions with a relative measure of impact, and researchers with a way to stratify these resources.8 When applied to medical journals, the Social Media Index correlates well with traditional measures of impact.8 The Social Media Index has been criticized because its score may be more related to popularity than quality.9 We sought to determine whether it could approximate the quality of blog posts by investigating its correlation with the mean gestalt, ALiEM AIR, and METRIQ-8 scores provided by a general population of medical raters. MATERIALS AND METHODS The METRIQ study is a cross-sectional rating study that evaluated 20 blog posts from 10 blogs, using 4 instruments: the Social Media Index score, gestalt rating, the ALiEM AIR score, and the METRIQ-8 score. Multiple analyses and publications, including this study, were planned a priori with the same data set including the 2 Annals of Emergency Medicine
recruitment methodology,10 reliability of the gestalt data,4 and reliability of the METRIQ-8 and ALiEM AIR scores.7 The METRIQ study protocol met the University of Saskatchewan’s Research Ethics Board requirements for exemption (BEH 16-09). On February 24, 2016, 10 blogs (Appendix E3, available online at http://www.annemergmed.com) were selected with a random-number generator (https://www. randomizer.org/) from a comprehensive list of emergency medicine and critical care Web sites that was created1 in 2013 and updated with a Google search in February 2016. In this update, one author (M.S.) reviewed the first 500 results for “emergency medicine” AND “blog” OR “podcast” and added Web sites written in English that had published a clinical blog post within the past 6 months. Web sites that had not published clinical blog posts since January 1, 2016, were excluded, along with blog posts consisting only of the “show-notes” of an accompanying podcast. In this review, one author (M.S.) identified Web sites for exclusion and another (B.T.) reviewed them. To avoid selection bias, each Web site’s 2 most recently published clinical blog posts were used. As described below and elsewhere in greater detail,10 a convenience sample of medical students, emergency medicine residents, and emergency medicine attending physicians was recruited through a multimodal recruitment strategy. A power calculation was initially conducted to guide enrollment for the gestalt analyses.4 However, because of the multiple complex analyses planned, the focus was on recruiting the maximum number of participants and incorporating the maximum feasible number of blog posts rather than a specific target. The recruitment strategy included in-person and e-mail communications with potential raters, as well as a social media strategy using Facebook and Twitter, a YouTube video, blog posts on the METRIQ Study Web site, and a podcast and blog hosted on the Skeptic’s Guide to Emergency Medicine. Raters who rated all blog posts are recognized as METRIQ study collaborators. Recruitment occurred between March 1 and April 30, 2016. The rating period was open between March 1 and June 1, 2016. Potential raters were directed to the METRIQ Study Web site (http://metriqstudy.org/) to complete a study intake form. A random-number generator (https://www.randomizer.org/) was used to generate blocks of numbers from 1 to 4 with no repeated values. Medical students, residents, and attending physicians were assigned to survey versions consecutively within separate blocks. Within 24 hours, these potential raters were e-mailed the information needed to consent to and enroll in the study. From that point until they completed rating, they were sent Volume
-,
no.
-
:
-
2018
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs
Figure 1. SMi formula for a given Web site. SMi, Social Media Index.
a reminder e-mail every 7 to 10 days a maximum of 4 times. In addition, a final notification was sent to all raters that the study was closing within 10 days. The online rating survey (Appendix E4, available online at http://www.annemergmed.com) was created with FluidSurveys.com. Four authors of the METRIQ study (including B.T., T.C., and M.L.) conducted an internal pilot of the survey that resulted in minor changes and estimated its length to be 90 minutes. This was thought to be too long for a single session, so the option to start and stop the survey at the raters’ convenience was added. The survey asked raters to rate each of the 20 blog posts for quality on a 7-point Likert scale, using gestalt. These data were collected alongside blog post evaluations with one or both of the ALiEM AIR and METRIQ-8 instruments.5,6 Each rater rated 10 blog posts with both instruments, 5 with only ALiEM AIR and 5 with only METRIQ-8. To minimize the effect that the ratings would have on gestalt evaluation, raters were instructed to provide their gestalt rating before using the instruments. Four versions of the survey were developed to minimize order effect and ensure that each blog post was rated approximately the same number of times with the ALiEM AIR and METRIQ-8 instruments. The Social Media Index score was calculated for the entire list of Web sites in February 2016, November 2016, April 2017, and December 2017,8 using the updated formula published on the ALiEM Web site (https://aliem. com/social-media-index). This version of the formula was modified after the previous publication8 because Google PageRank is no longer supported. Primary Data Analysis Descriptive statistics were calculated for the rater demographics and the ratings of each blog post. Raters who did not rate all assigned blog posts were excluded. Pearson’s correlations were calculated for the blog posts between the Social Media Index score (February 2016, November 2016, June 2017, and December 2017) and the average rater gestalt, ALiEM AIR, and METRIQ-8 scores of the raters. Volume
-,
no.
-
:
-
2018
Modified Bland-Altman plots were created to visually compare the mean Social Media Index score with mean rater gestalt rating, mean ALiEM AIR score, and mean METRIQ-8 score. For this comparison, each of the comparison variables was normalized on a 10-point scale. These figures are not traditional Bland-Altman plots because they compare aggregate variables and use mean values rather than individual data points. RESULTS After the Google search and application of the exclusion criteria, 60 Web sites were eligible for the study and 10 were selected. A total of 309 of the 330 potential raters who consented to the study completed all ratings (93.6%). Each blog post was rated with gestalt by 309 raters and using the ALiEM AIR and METRIQ-8 instruments by 229 to 236 raters. Raters were recruited from 27 countries, with the preponderance from North America. Most raters (75.0%) read blog posts for medical education purposes at least once per month. The raters’ demographics are presented in Table 1. With Pearson’s correlation calculations, the Social Media Index score correlated moderately to strongly with mean gestalt ratings (range 0.69 to 0.76) and moderately with mean ALiEM AIR scores (range 0.55 to 0.61) and METRIQ-8 scores (range 0.53 to 0.57). These correlations persisted for nearly 2 years after the month of the blog post’s publication (Table 2). Modified Bland-Altman plots visually representing the relationship between the mean Social Media Index with the normalized mean gestalt, ALiEM AIR score, and METRIQ-8 score are presented in Figure 2. The y axis displays the difference between the Social Media Index score and the normalized score for each blog, whereas the x axis displays the average of the Social Media Index score and the normalized score for each blog post. The negative average value for each score on the y axis indicates that the Social Media Index generally underestimates a blog post’s quality relative to the other metrics. This effect is more pronounced for lower-quality blog posts. Annals of Emergency Medicine 3
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs Table 1. Rater demographics and characteristics (n ¼ 309). Variable
No. (%)
Level of training Medical students
121 (39.2)
Emergency medicine residents
88 (28.5)
Emergency medicine attending physicians
100 (32.4)
Age, mean (SD), y
31.1 (7.3)
Sex Male
184 (59.5)
Female
123 (39.8)
Other
2 (0.6)
How often do you read blog posts on medical topics for educational purposes? Every day
48 (15.5)
Several times a week
141 (45.6)
Once a week
43 (13.9)
Several times a month
38 (12.3)
Once a month
21 (6.8)
Less than once a month
15 (4.9)
Never
2 (0.6)
No response
1 (0.3)
Do you manage, edit, own, or operate a blog that posts on medical topics for educational purposes? Yes
45 (14.5)
No
261 (84.5)
No response
3 (0.6)
Country Canada
146 (47.2)
United States
111 (35.9)
Other (25 countries)
49 (15.9)
LIMITATIONS First, there is no universally recognized definition of quality with which the Social Media Index may be compared. We addressed this by including the reference standard of the average gestalt of many raters,4 along with the only 2 quality evaluation instruments that have been derived to appraise blog posts targeting clinicians.5,6 Lacking true values for quality, these scores provide multiple quantitative perspectives to provide as comprehensive an evaluation of the
Social Media Index as possible. Second, previous studies have demonstrated that gestalt has limited application because large numbers of raters must be used to obtain reliable values.4 However, enough raters were included in this study to ensure an accurate comparison. Third, raters were not blinded to the Web site domain for the featured blog posts. This may have resulted in their rating familiar or high-profile blogs more favorably than others. Fourth, the dates that the Social Media Index score was calculated were not standardized. Because this would be more likely to increase the variability in Social Media Index calculations because of annual variations in viewership, it is unlikely to have led to spurious correlations. Fifth, only 2 blog posts from a single month were evaluated, which may not have been representative of all blog posts on a Web site and could have skewed the quality ratings of some Web sites, affecting their correlation with the Social Media Index in an undetermined direction. DISCUSSION Quality assessment of online resources is challenging, especially for individuals and learners. High number of raters are required to generate reliable gestalt ratings4 and the AIR score has been studied only in a group of trained attending physicians.5 The METRIQ study was conducted to test our hypothesis that Web sites identified by the Social Media Index as being more influential would also be of higher quality.8 Our results demonstrated a consistent correlation between the Social Media Index score and the average ratings of blog posts from a large group of raters using gestalt and 2 quality evaluation instruments longitudinally over time. Because a criterion standard for quality in medical education blogs is lacking, quality was assessed from the perspectives of all 3 of the approaches published in the literature. As the ratings used in this data set have demonstrated reliability, they can be assumed to reflect an accurate quality assessment.4,7 These findings suggest that the Social Media Index score can be an important surrogate marker of quality that can guide individuals and small groups who would otherwise be unable to reliably assess resource quality.
Table 2. Pearson’s correlations between the Social Media Index score, gestalt, ALiEM AIR score, and METRIQ-8 score. SMi (95% CI) February 2016
November 2016
April 2017
December 2017
Range
Gestalt
0.69 (0.34–1.00)
0.72 (0.38–1.00)
0.70 (0.35–1.00)
0.76 (0.44–1.00)
0.69–0.76
ALiEM AIR
0.55 (0.13–0.96)
0.56 (0.15–0.97)
0.55 (0.13–0.96)
0.61 (0.21–1.00)
0.55–0.61
METRIQ-8
0.53 (0.12–0.95)
0.56 (0.15–0.97)
0.52 (0.10–0.95)
0.57 (0.16–0.97)
0.53–0.57
CI, confidence interval.
4 Annals of Emergency Medicine
Volume
-,
no.
-
:
-
2018
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs
does correlate with multiple interpretations of quality and that the relationship between quality and the Social Media Index score is persistent. The correlation between the Social Media Index and quality assessments makes sense. Web sites with lowerquality content may be less likely to be viewed, shared, and “liked” by clinicians seeking reliable resources. We suspect that clinicians are also less likely to consistently visit Web sites that they think are of lesser quality. Thus, the Social Media Index score may serve as a crowdsourced gestalt assessment of the quality of a Web site’s longitudinal content. However, Figure 2 demonstrates that, relative to the quality evaluations, the Social Media Index underestimates the quality of lower-scoring resources. This suggests that there may be high-quality resources that are not identified by the Social Media Index because they are less influential. Consideration was given to correlating subgroups of raters (medical students, emergency medicine residents, and emergency medicine attending physicians) with the Social Media Index separately. However, because the average ratings of these groups was so strongly correlated (Pearson’s correlation >0.92 between the subgroups for all 3 ratings),4 the subgroup results were not expected to differ from those of the larger group. The Social Media Index score for educational Web sites in emergency medicine and critical care was shown to correlate reasonably well with quality evaluations of blog content. Although critical appraisal of article-level online content is still warranted, this study supports the use of the Social Media Index as a tool to guide individuals and groups toward high-quality social media Web sites. Supervising editor: N. Seth Trueger, MD, MPH
Figure 2. Modified Bland-Altman plots comparing the mean SMi value with the normalized mean gestalt rating, mean ALiEM AIR score, and mean METRIQ-8 score for each blog post.
Although the Social Media Index score correlated reasonably well with gestalt, ALiEM AIR, and METRIQ-8, its correlation with gestalt was strongest, possibly because certain aspects of raters’ preferences are more aptly captured by gestalt (eg, design, layout) than criterion-based scores. Moreover, there may have been an inevitable halo effect when raters evaluated blog posts on more “popular” or influential Web sites as having higher quality because of greater familiarity with these educational resources. A consistent positive correlation was found between the Social Media Index score and both criterion-based tools over time. This suggests that the Social Media Index score Volume
-,
no.
-
:
-
2018
Author affiliations: From the Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada (Thoma, Kapur, Siemens); Health Professions Education, Massachusetts General Hospital Institute, Boston, MA (Thoma); the Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, Ontario, Canada (Chan); Wayne State University, Detroit, MI (Sifford); the Department of Emergency Medicine, Regions Hospital, HealthPartners Institute, Saint Paul, MN, and the Department of Emergency Medicine, University of Minnesota, Saint Paul, MN (Paddock, Ankel); the HealthPartners Institute, Saint Paul, MN (Ankel); the USCþLAC Department of Emergency Medicine and David Geffen School of Medicine at UCLA, Los Angeles, CA (Grock); and the Department of Emergency Medicine, University of California–San Francisco, San Francisco, CA (Lin). Author contributions: BT, TC, PK, DS, MS, and ML conceived the study and designed the trial. BT obtained research funding, supervised the conduct of the trial, and drafted the article. BT, PK, Annals of Emergency Medicine 5
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs and MS performed data collection. BT, TC, MS, MP, FA, AG, and ML recruited raters. TC provided statistical advice on study design and analysis. BT and TC analyzed the data. All authors contributed substantially to article revision. BT takes responsibility for the paper as a whole. All authors attest to meeting the four ICMJE.org authorship criteria: (1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND (2) Drafting the work or revising it critically for important intellectual content; AND (3) Final approval of the version to be published; AND (4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org). Funding for this research was provided by the Canadian Association of Emergency Physicians (Junior Investigator Grant) and the Royal College of Physicians and Surgeons of Canada (Robert Maudsley Fellowship for Studies in Medical Education). The following authors report operating medical education blogs: Brent Thoma (ALiEM.com, CanadiEM.org, Debrief2Learn.org), Teresa Chan (ALiEM.com and CanadiEM.com), Derek Sifford (ALiEM.com), and Michelle Lin (ALiEM.com). Publication dates: Received for publication January 9, 2018. Revisions received February 9, 2018, and March 31, 2018. Accepted for publication May 1, 2018.
REFERENCES 1. Cadogan M, Thoma B, Chan TM, et al. Free open access meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002-2013). Emerg Med J. 2014;31:e76-e77. 2. Mallin M, Schlein S, Doctor S, et al. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014;89:598-601. 3. Cameron P. Pundit-based medicine. Available at: http://www. epijournal.com/articles/240/pundit-based-medicine. Accessed January 15, 2016. 4. Thoma B, Sebok-Syer SS, Krishnan K, et al. Individual gestalt is unreliable for the evaluation of quality in medical education blogs: a METRIQ study. Ann Emerg Med. 2017;70:394-401. 5. Chan TM, Grock A, Paddock M, et al. Examining reliability and validity of an online score (ALiEM AIR) for rating free open access medical education resources. Ann Emerg Med. 2016;68:729-735. 6. Chan T, Thoma B, Krishnan K, et al. The derivation of two simplified critical appraisal scores for use by trainees to evaluate online educational resources: a METRIQ study. West J Emerg Med. 2016;XVII:574-584. 7. Sebok-Syer SS. Colmers-Gray I, et al. Quality evaluation scores are no more reliable than gestalt in evaluating the quality of emergency medicine blogs: a METRIQ study. Teach Learn Med. 2018;30:294-302. 8. Thoma B, Sanders JL, Lin M, et al. The Social Media Index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015;16:242-249. 9. Carley S. The Social Media Index (SMi): can and should we measure FOAMed? St Emlyn’s Blog. Available at: http://stemlynsblog.org/ the-social-media-index-smi-is-it-flawed/. Accessed April 11, 2016.
6 Annals of Emergency Medicine
10. Thoma B, Paddock M, Purdy E, et al. Leveraging a virtual community of practice to participate in a survey-based study: a description of the METRIQ study methodology. AEM Educ Train. 2017;1:110-113.
APPENDIX The METRIQ Study Collaborators (in alphabetical order) Charlotte Alexander, Mohammed Alkhalifah, Abdulaziz S. Almehlisi, Saeed Alqahtani, Scott Anderson, Shelaina Anderson, Colin Andrews, Jocelyn Andruko, Felix Ankel, Nikytha Antony, Diptesh Aryal, Barbra Backus, Jennifer Baird, Andrew Baker, Sarah Batty, Jared Baylis, Braeden Beaumont, Chris Belcher, Brent Benavides, Michael Benham, Julian Botta, Nicholas Bouchard, Victoria Brazil, Emily Brumfield, Anthony Bryson, Wisarut Bunchit, Kat Butler, Lindy Buzikievich, David Calcara, Rob Carey, Stephen Carroll, Louise Cassidy, Kirsty Challen, Kathryn Chan, Tim Chaplin, Natasha Chatham-Zvelebil, Eric Chen, Lucy Chen, Sushant Chhabra, Alvin Chin, Eric Chochi, Tina Choudhri, Jeremy Christensen, Isabelle Colmers-Gray, Kimberly Connors, Veronica Coppersmith, Abby Cosgrove, Gregory Costello, Kevin Cullison, Andrew D’Alessandro, Kerstin de Wit, Marie Decock, Rayan Delbani, William Denq, Julianna Deutscher, Brendan Devine, Maia Dorsett, Taylor Duda, Justin Dueweke, Teresa Dunphy, Sean Dyer, Karthryn T Eastley, Marcia Edmonds, Ken Edwards, Robert Ehrman, Youness Elkhalidy, Preston Fedor, Brian Ficiur, Caley Flynn, Bill Fraser, Meagan Fu, James Fukakusa, Eric Funk, Damjan Gaco, Viktor Gawlik, Kenn Ghaffarian, Laleh Gharahbaghian, Phil Griffith, Andrew Griffith, Andrew Grock, Tanner Gronowski, Cathy Grossman, Jaroslaw Gucwa, Pawan Gupta, Alexandra Gustafson, Andrew Guy, Mary Haas, Stanislaw Haciski, Emina Hajdinjak, Andrew K. Hall, Regina Hammock, Jan Hansel, Alexander Hart, Larissa Hattin, Brandon Herb, SueLin Hilbert, Jesse Hill, Jeff Hill, Amy Ho, Emily House, Nina House, James Huffman, Charlie Inboriboon, Alex Ireland, Ali Jamal, Mohammad Ali Jamil, Victor Jansen, Zach Jarou, Vivian Jia, Levi Johnston, Drew Kalnow, Puneet Kapur, Seth Kelly, Kyle Kelson, William Kent, Rishi Khakhkhar, Jaasmit Khurana, Ashley Kilp, Scott Knapp, Sebastian Kohler, Ivanna Kruhlak, Nadim Lalani, Samantha Lam, Patrick Lank, Zander Laurie, Kristina Lea, Ernest Leber, Ching-Hsing Lee, Haakon Lenes, Nilantha Lenora, Jesse Leontowicz, Kelly Lien, Yingchun Lin, Michelle Lin, Andrew Little, Ivy Liu, Harry Liu, Steve Liu, Stephanie Louka, Elise Lovell, David Lowe, Ashley Lubberdink, Jessica Luc, Casey Lyons, Sheng-Hsiang Ma, Hugh MacLeod, Nick Mancuso, Anali Maneshi, Jesse May, John Mayo, Mike McDonnell, Susan McLellan, Carolyn Volume
-,
no.
-
:
-
2018
Thoma et al
Social Media Index as an Indicator of Quality for Emergency Medicine Blogs
McQuarrie, Julia Nood, Therese Mead, Cory Meeuwisse, Patrick Meloy, Perry Menzies, Anne Messman, Stephen Miazga, Logan Mills, Ken Milne, Allan Mix, Steve Montag, Brendan Moore, Justin Morgenstern, Sarah Mott, P. Mukherj, Ali Mulla, Sheena Nandalal, Taylor Nikel, Sean Nugent, Morgan Oakland, Werner Oberholzer, Onyeka Otugo, Taofiq Segun Oyedokun, Mike Paddock, Alim Pardhan, Kinjal Patel, Quinten Paterson, Catherine Patocka, Christine Patterson, James Pearlman, Elyse Berger Pelletier, Alexis Pelletier-Bui, Marc Phan, Zafrina Poonja, Aubrey Powell, Kamini Premkumar, Gregor Prosen, Vishal Puri, Tanis Quaife, Ryan Raffel, Ali Raja, Randi Ramunno, Louise Rang, Suzanne Rannazzisi, Shauna Regan, Salim R. Rezaie, Milan Ridderikhof, Vanessa Rogers, Christine Roh, Dra. Maria Rosa Carrillo, Keith Rosenberg, Marina Roure, Sherri Rudinsky, Joshua Rudner, Adeeb Saleh, Will Sanderson, Owen Scheirer, Paul Schofield, Paul Schunk, Evan Schwarz, Parisa Shahrabadi, Eric Shappell, Julia Sheffield, Jonathan Sherbino, Manpreet Singh, Hector C
Volume
-,
no.
-
:
-
2018
Singson, Dave Slessor, Sam Smith, Paula Sneath, Robert Sobehart, Kerry Spearing, James Stempien, Britni Sternard, Tara Stratton, Katherine Stuart, Bob Stuntz, Michael Susalla, Colleen Sweeney, Loice Swisher, Henry Swoboda, Shahbaz Syed, Taku Taira, Nikhil Tambe, Richard Tang, Elisha Targonsky, Rachel Taylor, Alan Taylor, Todd Taylor, Paxton Ting, Gerhard Tiwald, Kelvin Tran, Evelyn Tran, Jason Trickovic, Paul Trinquero, Seth Trueger, Aaron Tyagi, Manrique Umana, Patrick Vallance, Patricia Van den Berg, Luis Vargas, Rene Verbeek, Sandra Viggers, Zlata Vlodaver, Matthew Wagner, Noorin Walji, Joe Walter, Miranda Wan, Rachel Wang, Gregory Wanner, Wyatt Warawa, Mike Ward, Jennifer Weekes, Kristen Weersink, Cara Weessies, Anna Whalen-Browne, Brian Whiteside, Matthew Willis, Jonas Wilmer, Nelson Wong, Mark Woodcroft, Rob Woods, Lawrence Yau, Jessica Yee, Calvin Yeh, Simon York Ming Huang, Katherine Yurkiw, Fareen Zaver, and Alexander Zozula
Annals of Emergency Medicine 7