- Email: [email protected]
Examining internet resources on gender differences in ACL injuries: What patients are reading
The Knee 20 (2013) 196–202
Contents lists available at SciVerse ScienceDirect
The Knee
Examining internet resources on gender differences in ACL injuries: What patients are reading Michelle M. Gosselin a,⁎, Mary K. Mulcahey b, Edward Feller c, Michael J. Hulstyn b a b c
Warren Alpert Medical School of Brown University, United States Department of Orthopaedics, Warren Alpert Medical School of Brown University, United States Department of Community Health, Warren Alpert Medical School of Brown University, United States
a r t i c l e
i n f o
Article history: Received 29 June 2012 Received in revised form 31 October 2012 Accepted 10 November 2012 Keywords: Anterior cruciate ligament (ACL) injuries Internet information Gender differences Female ACL
a b s t r a c t Background: The Internet is a popular and powerful resource used by both clinicians and patients. However, medical information on the Internet is often unregulated, biased, and of poor quality. Given the frequency of ACL injuries in young women, this topic was used as a case study to examine the information available to patients on the Internet. Methods: An Internet search was performed using the top three general search engines for four different search phrases related to ACL injuries in females. The top ten sites from each search were analyzed on the basis of both website interface and a specific content scoring guide. Results: A total of 35 unique websites were analyzed. The majority of websites were authored by a layperson or had an unidentified author. Readability data indicated that many sites were written at literacy levels that were too difficult for the average reader. Finally, the average content analysis score for all sites was 41.6 out of a possible 100 points (range 10–87.5); only one-third of the websites cited references for the information provided. Conclusions: We found the Internet information on ACL injuries in women to be largely unsupported, outdated and too difficult for the average reader to comprehend. The average site covered less than half of the topics that we considered relevant to ACL injuries in females indicating that information available on the Internet is largely incomplete. Clinical relevance: Clinicians should be aware of the deficiencies in Internet information in order to appropriately address these issues with patients. © 2012 Elsevier B.V. All rights reserved.
1. Introduction 1.1. Internet use The Internet has become an integral part of our society. In 2011, it was estimated that approximately 80% of the United States population, or more than 270 million people, used the Internet [1]. Further, roughly 2.5 billion people worldwide used the Internet in 2011, indicating that the Internet is rooted in cultures all over the globe [2]. The Internet continues to revolutionize the way people communicate and obtain information. It has been found that searching for health information using an Internet general search engine, such as Google, is the third most common use of the Internet behind email and product research [3]. The DECISIONS study revealed that patients rated information they received from the Internet second only to information they received from their healthcare provider [4]. This suggests that patients not only access the Internet for information regarding their ⁎ Corresponding author at: Box G-9045, 222 Richmond Street, Providence, RI 02903, United States. Tel.: +1 203 640 5521; fax: +1 401 444 3609. E-mail address: [email protected] (M.M. Gosselin). 0968-0160/$ – see front matter © 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.knee.2012.11.006
health, but that they accept what they read as fact. The unregulated nature of the Internet, allows anyone to publish health related information. This reality puts patients at high risk of making decisions about their health based on poor quality information. 1.2. Quality of online materials Several studies have assessed Internet use and the quality of health information available online related to specific conditions such as scoliosis, carpal tunnel syndrome, and a variety of sports medicine diagnoses [5–7]. Generally, the studies revealed poor quality and accuracy of webbased information related to these topics. An evaluation of Internet information regarding carpal tunnel syndrome revealed that 23% of the sites offered unconventional or misleading information [5]. Similarly, a study examining websites related to scoliosis found that 90% of the websites scored less than 16 out of a possible 32 points for content quality and only 3 of 50 sites examined were between 76 and 100% accurate [6]. With increasing awareness of the deficiencies of web-based health information, many organizations have attempted to create specific standards or criteria to evaluate website content and guide patients
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
on how to critically read these sites. The Health on the Net (HON) Foundation and the American Medical Association (AMA) are among the many groups attempting to establish a set of guidelines to ensure that the information read by consumers meets certain standards [8,9]. Websites can receive certification from these organizations indicating that their website upholds the criteria established by that organization. A review of Internet information on ten common orthopedic sports medicine diagnoses found a correlation between presence of HON seal and greater content and website credibility scores [7]. However, these principles go largely unused by both health information consumers and website developers; only 37 of the 154 (24%) websites examined in this study had the HON seal [7], increasing the risk of unregulated, inaccurate and outdated Internet health information. Injury to the anterior cruciate ligament (ACL) is a very common and costly problem, especially among young and otherwise healthy athletes. It is estimated that between 80,000 and 250,000 ACL injuries occur in the United States each year with more than 50% of these occurring in athletes aged 15–25 years old [10]. This age group comprises 45% of Internet users making them especially susceptible to reading information about ACL injuries online [2]. Given the financial and individual costs of this condition, ACL injuries affect many people in the patient's life. We speculate that parents, coaches, athletic trainers and physical therapists likely turn to the Internet for information regarding injuries like an ACL tear as patients often encounter these people prior to seeing an orthopaedic surgeon. Using ACL injuries in women as a case study, we evaluated general trends in the Internet resources that patients, parents, coaches or athletic trainers might access for information regarding this condition. To our knowledge, there is no previous literature on this subject. The goal of this study is to critically assess the quality of the information regarding ACL injuries in women available to patients on the Internet.
2. Materials and methods 2.1. Search methodology An algorithm illustrating our search methods is shown in Fig. 1. We performed a search using selected keywords on the top three general search engines: Google, Yahoo and Bing [11]. The Yahoo search engine was powered by Bing, so effectively there were two separate sets of results for each search term since the Yahoo and Bing searches yielded the same set of websites. Data from December 2011 revealed that Google was used for 66.6% of all general Internet searches in the US, followed by 16% on Yahoo and 12% on Bing [12], indicating that we examined 95% of the general searches the average individual performed. We chose to use general search engines, as opposed to a medical information specific search, since patients more often turn to these general searches when looking for health information. To control for the variation in keywords searched by patients, we used four separate, but similar, search terms: “ACL injuries in women”, “ACL injuries in female athletes”, “knee injuries in women”, and “knee injuries in female athletes”. Searches performed using the more general phrases “ACL injuries” or “knee injuries” yielded a large proportion of news articles regarding professional athletes who
197
had recently suffered knee injuries. We did not believe this would be a fruitful search topic and thus narrowed the search terms to address the gender differences in ACL/knee injuries. All searches were performed on February 10, 2012 to account for the day-to-day variation in the top ten sites returned with each search query. Data regarding number of hits for each search term on the three different search engines is reported in Table 1. Most people do not look beyond the first page of search results that lists the top websites for a given search [13]thus we chose to evaluate only the top ten search results for each query. The websites were sorted; all duplicates and non-functioning website links were removed to compile a master list of unique websites to be analyzed (Appendix A). We did not exclude any websites on the basis of content or design, as we sought to evaluate the webpage that any given patient may read after performing a general search. 2.2. Data evaluation The website interface was assessed for general information regarding the site design, authorship and readability. Website content was also evaluated on the basis of completeness, validity, and supporting references specific to gender differences in ACL injuries. 2.3. Authorship The websites were first classified into one of seven categories: physician, non-physician, academic, commercial product, commercial site, layperson or unidentified. We used the following definitions, similar to that previously described [5]: physician: author or authors were individual or group physicians with no research or university group, non-physician: all alternative medical providers including chiropractors, physical and occupational therapists, academic: author or authors were affiliated with a university or research group, commercial site: the author represented a commercial web site without an interest in a specific commercial product, commercial product: an author or authors who were marketing a specific commercial product related to ACL injuries or prevention, layperson: individuals or organizations who did not belong to any of the previous categories and who maintained a noncommercial web site for providing information about knee injury, or unidentified: the author, or author's credentials, were not specified. Two examiners categorized each site independently; in cases of disagreement, a third examiner cast the deciding vote. We also sought to determine whether a third-party group (e.g. Health on the Net [HON] Foundation or the American Medical Association [AMA]) certified the websites as indicated by the presence of the entity's seal. 2.4. Readability analysis We assessed the readability of the website text using the Flesch– Kincaid grade and the Flesch reading ease tools [3,14]. This assessment was performed using the readability statistics function in Microsoft Word 2007. The health information readability goals, Flesch–Kincaid
Fig. 1. Algorithm for the search methodology. This algorithm illustrates the step by step website search process to show how we arrived at our final database of websites for evaluation.
198
M.M. Gosselin et al. / The Knee 20 (2013) 196–202 Table 1 This table lists the number of hits obtained with each search query on each of the general search engines used. Search query “ACL injuries in women” Google Yahoo Bing “ACL injuries in female athletes” Google Yahoo Bing “Knee injuries in women” Google Yahoo Bing “Knee injuries in female athletes” Google Yahoo Bing
# of hits 164,000 909,000 847,000
35,700 194,000 183,000
3,050,000 14,400,000 19,600,000
104,000 3,230,000 3,220,000
grade goal of b8 and the Flesch reading ease goal of > 60, have been set based on current literacy levels in the United States [14].
Table 3 Website content parameters. This table outlines the content scoring guide used to evaluate each website's content. The specific topics we examined are listed in the left column and the number of points allotted to each topic is noted in the right column; a total of 100 points were possible. Specific topic Knee anatomy Gender differences Gender difference statistics Hormonal differences Narrow intercondylar notch Knee angle Decreased strength (quadriceps/hamstring ratio) Neuromuscular factors Mechanism of injury Contact Non-contact Diagnosis: physical exam, imaging Treatment options Operative treatment Different graft options Non-operative treatment Treatment complications/long-term sequelae Addresses Prevention Strategies Braces Pre-competition Total
Points 5 5 5 5 5 5 5 5 5 10 5 5 5 15 5 5 5 100
3. Results
2.5. Website interface criteria Each individual site was further evaluated on the basis of the criteria listed in Table 2, with the goal of assessing the set-up, accessibility, and currency of the information contained on the website. These criteria were developed using the concepts outlined by the HON guidelines.
2.6. Website content analysis Content analysis of each website was performed using a point system developed by the authors (Table 3). Each site was evaluated for the presence of specific topics relevant to gender differences in ACL injuries including knee anatomy, potential reasons for female susceptibility to ACL injuries, diagnosis, treatment, and prevention strategies. The website was given the specified number of points if that topic was covered within the text of the site. Each website was evaluated and scored separately by two independent examiners. The content score was calculated by averaging the scores of two independent examiners and the inter-observer variability was noted. We also assessed whether each topic was supported by outside references and analyzed the number, quality, and currency of references provided.
After excluding all duplicate and non-functioning links, a list of 35 unique websites was compiled for analysis (Appendix A). 3.1. Website interface evaluation Only 18 of 35 websites (51%) provided a date of creation or date of most recent update; of those 18 sites, only three were updated within the year prior to the search. Sixteen websites (46%) had advertisements present and 12 websites (35%) had evidence of sponsorship, while 12 websites (35%) had evidence of a commercial intent, all of which had links to said commercial interest. These sites were often physician or non-physician authored sites with links to their practice, hospital or clinic, without explicit marketing intent. Only two websites (6%) had a third party seal (e.g. HON, AMA, etc.) to signify the certification by one of the organizations established to uphold a standard of health information on the Internet (Fig. 2). 3.2. Authorship The percentages of each category of authorship are shown in Fig. 3. The majority of the websites were authored by a layperson (37%) or the author was unidentified (23%). Ten websites (28%) were either authored by a physician or someone affiliated with an academic institution (i.e. a researcher or other healthcare provider). Nine of the
Table 2 Website interface criteria. This table lists the criteria used to evaluate each website on the basis of website layout and general set-up. Date of last update present Copyright Notice Statement of purpose Disclosure of authorship Author credentials Financial disclosure Presence of ads Presence of sponsorship Privacy statement Subscription/log in Contact information Commercial interest Links to commercial interest Third party seal (i.e. HON, AMA, etc.)
Fig. 2. Website interface characteristics. This graph illustrates the number of websites that exhibited each of the search parameters.
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
199
Table 4 Content analysis by website category. This table shows the average content score arranged by website category. Category
# of websites
(0–100)
Flesch–Kincaid grade (goal b8)
Flesch reading ease (goal >60)
Academic Physician Non-physician Commercial product Lay Unidentified Total
3 7 2 2 13 8 35
55.0 49.3 53.8 32.5 37.9 35.3 41.6
12.7 10.7 12.3 12.9 11.5 12.3 11.7
37.8 52.0 44.5 38.5 49.3 45.9 47.4
Fig. 3. Categorization of websites. This graph shows the percentage of websites that were classified in each category. The majority (60% of the total number of websites assessed) were classified as either lay person or the author was unidentified. Conversely, less than one-third (28% of the total number of websites) were authored by a physician or someone affiliated with an academic institution. Note: 0 websites were classified as “Commercial Site” so this category was omitted from the graph.
met the Flesch–Kincaid Grade goal and five met the Flesch Readability Score goal). Further, there was no correlation between readability scores and content scores of the websites we examined. Table 4 presents the average readability statistics broken down by website category.
websites (26%) were articles from a newspaper or other news outlet (i.e. television, radio) and three of the websites (9%) were blogs. Only two sites (6%) were classified as selling a specific commercial product, while no site was classified as a commercial site explicitly. This finding was different from results published on a similar search on “carpal tunnel syndrome”, which revealed that one-third of the websites endorsed a commercial product [5].
The content analysis results are presented in Fig. 5. The informational content scores were calculated for each website by two independent evaluators and the scores were averaged to give a single content score for each site. The mean informational content score for all sites was 41.6 out of a possible 100 points (range: 10–87.5). The mean informational content score broken down by website category is presented in Table 4. The academic, non-physician and physician classified sites had higher average content scores (55.0, 53.8 and 49.3 respectively) compared to the layperson, unidentified and commercial product sites (37.9, 35.3 and 32.5 respectively). Further, the intra-observer reliability of the informational content score was assessed. The mean difference in scores between the two observers was 1.4 points. Eighty-five percent of the websites were scored within five points of each other. Diagnosis, treatment and long-term sequelae were discussed on less than half the websites, most of which were physician or academic in classification. Gender difference statistics, risk factors and prevention strategies were discussed on 80% of the sites. Anatomy was well described by all site classifications except for layperson authored websites. Interestingly, only 12 of the 35 websites provided references for the information that was presented (Fig. 6). All academic (3), non-physician (2) and commercial product (2) classified sites provided references for the information presented on their websites.
3.3. Readability analysis The readability scores for 34 sites are shown in Fig. 4. One website was in a PDF format and could not be copied into Word 2007 to complete the readability statistics assessment. The mean Flesch–Kincaid Grade (goal b8) for all 34 websites was 11.77 ± 2.13 and the mean Flesch Readability Score (goal >60) was 47.4 ± 10.87; both scores are significantly above goal reading levels for health information for the general population based on previous studies [3,14]. Ninety-one percent of websites were too difficult according to the Flesch–Kincaid Grade goal of b8 and 86% were above the goal reading level according to Flesch Readability Score goal of >60 (only three websites
3.4. Content analysis
Fig. 4. Website readability. This graph depicts the Flesch–Kincaid grade and the Flesch reading ease scores of each website analyzed. The red lines indicate the goal reading levels of health materials (Flesch–Kincaid grade (goal of b8) and the Flesch reading ease (goal of >60)) given the literacy of US adults. Less than 5 websites achieved these goals indicating that, overall, the websites were too difficult for the average US adult to fully comprehend.
200
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
Fig. 5. Website content analysis scores. This graph illustrates the content analysis scores obtained by two independent reviewers, broken down by website. Both scorers scored websites consistently throughout with an average difference between the two reviewers of 1.4 points. The final column shows the average content score of all websites examined for the two reviewers.
Among the 12 websites that provided references, many of those references were outdated, incomplete or contained non-functional links. The three academic sites provided the greatest number and most up-to-date and complete set of references of these 12 sites; otherwise, there was no correlation between presence of references and overall content score.
4. Discussion Several studies have examined Internet information regarding various orthopedic conditions [5–7]. Many of these studies have found that information online is outdated, unsubstantiated, and often has a commercial interest present. Further, content analyses have shown that the information is largely incomplete and in some cases, inaccurate [5–7]. There is no standardized way of regulating the type or quality of information available on the Internet, so physicians need to be aware of the potential problem of misinformation available online. Our findings concur with literature available regarding medical information on the Internet. In our study, 37% of the websites identified were authored by a layperson and 23% had unidentified authors, indicating that the majority of sites the health information consumer reads are either written by someone outside the medical profession or the author is not disclosed. Yet, health consumers generally believe what they read on the Internet without questioning the source of the information. Our study revealed that many websites were outdated, only 3 with evidence of having been updated within the last year.
Fig. 6. References provided by each website. This graph shows the number of websites that provided a given number of references for the information presented on their site. As can be noted, the majority of the websites did not provide any references while very few provided more than 5 references.
Given that 12 of 35 websites analyzed showed some form of commercial interest, the intent and motives behind the information supplied can be called into question. It is possible that the website sponsor or author is providing biased information as a means of promoting a specific product, practice or service. The lack of current, high-quality references also questions the validity and accuracy of information on many sites. Most websites did not cite specific references and on those websites where references were provided, many were outdated and of poor quality. Unlike physician disclosure requirements, these websites are not required to disclose any financial or commercial relationships, as the sites are largely unregulated. Most institutions, societies and journals require physicians and researchers to disclose any commercial relationship or potential conflict of interest. One of the goals of the Disclosure Program of the AAOS is to: increase transparency throughout AAOS by making this disclosure program available to AAOS members and the public [15]. The AMA and HON have attempted to improve the transparency and accountability of these websites through their established guidelines, but studies have indicated that these programs are underutilized. The HON website notes that 7300 websites are registered and certified through their program, while there are over 50,000 websites containing medical information [8], implying that less than 15% of websites are HON certified. The fact that only two websites (6%) in our study had certification by a third party raises several concerns. First, it is possible that website designers are not aware of the possibility of obtaining a “seal of approval” to indicate that the site has met certain accepted standards. Second, it could illustrate that the websites accessed by health consumers do not uphold the credible guiding principles of these organizations. Regardless, there should be greater standardization of certification modalities. Numerous organizations have established guidelines for health information on the Internet; information regarding the third party seal could be more efficiently distributed if there was only one organization governing the standards. Further, it would be easier to educate patients regarding one set of principles to help them more effectively, and critically, read health information on the Internet. Our results raise another concern not addressed in similar studies: the readability of most of the online health information we examined was far too difficult for the average American consumer. It has been reported that ninety-two million Americans have less than eighthgrade level reading skills [14], yet the average reading level of the websites we examined was above the eleventh-grade level. Less than ten percent of the websites met both the Flesch–Kincaid Grade
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
and the Flesch Readability Score goal levels. This indicates that many patients, parents, coaches or trainers may not be able to fully comprehend the information read while performing an online search. As the field of medicine continues to move toward a shared decision making model, patient's comprehension of medical information is becoming increasingly important. It has been shown that increasing readability leads to improvement in understanding and often also improved cooperation with treatment [16]. Further, it is thought to be less important that patients have complete knowledge recall regarding a particular topic, but that they conceptually understand their disease, injury or treatment plan so that they may play an active role in decision making [17]. Improving readability of patient information has the potential to improve patient satisfaction and ultimately patient care. Finally, we analyzed the topics discussed on the websites accessed with simple general search engine queries. Since we focused our search terms on women/females, it was not surprising that our searches turned up articles focused on gender differences in ACL injuries as well as possible prevention strategies. All but two websites discussed at least one of the risk factors associated with an increased risk of knee injury in women. We found that many websites lacked information on the signs of a serious injury, when to see a doctor, possible treatment options as well as long-term problems that could arise from this injury. Our study has several limitations. We used only the top three general search engines and did not include any medical specific search engines (i.e. Medline, WebMD). These three search engines, however, comprised 95% of the general searches performed by laypeople [12]. Further, we examined only the top ten websites from each search query, which gave us a limited number of websites to evaluate. However, given previous data regarding consumer Internet habits, we felt that this search strategy was most similar to that used by patients when researching health information online and gave us a total number of websites similar to previous studies. Additionally, we used a very specific search topic. It is possible that other orthopedic topics are covered in more detail on the Internet. We limited our search to the English language; searches using other languages may yield different results. Finally, our content analysis simply acknowledged the presence of specific information regarding a myriad of topics related to ACL/knee injuries in women. We attempted to make our grading scale standardized and objective, however, the possibility of observer bias in examiners' website grading cannot be excluded. In conclusion, we found the Internet information on ACL injuries in women to be largely unsupported, outdated and too difficult for the average reader to comprehend. The average site covered less than half of the topics that we considered relevant to ACL injuries in females indicating that information available on the Internet is largely incomplete. Internet website authors should focus on readability of patient information, as this has the potential to improve patient satisfaction and ultimately patient care. Future studies should be aimed at developing a set of strategies that consumers could employ to more effectively and critically read information they access on the Internet. Conflict of interest statement None of the authors have any financial or personal relationship with other people or organizations that may bias this study. Acknowledgments All authors made significant contributions to the development of the study design, completion of study, as well as preparation and editing of the manuscript; there were no other contributors to the study. This study has no source or funding and thus has no study sponsor involvement.
201
Appendix A. Listing of websites evaluated
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
35
http://www.youcanbefit.com/ACL.html http://articles.cnn.com/2007-03-26/health/hm.acl.girls_1_acl-injuriesfemale-athletes-knee-braces?_s=PM:HEALTH http://www.hughston.com/hha/a_11_3_2.htm http://www.thesportjournal.org/article/prevention-acl-injuries-femaleathletes-through-early-intervention http://orthopedics.about.com/od/aclinjury/tp/acl.htm http://media.ncaa.org/Skins/ACLSkin/Default.aspx?PageId=6c28f11c-85a74a22-911d-41e934e98e6b&ss=MediaPortal http://www.momsteam.com/health-safety/muscles-joints-bones/knee/aclinjuries-in-female-athletes http://bjsm.bmj.com/content/42/6/394.full http://www.cdc.gov/media/pressrel/2008/r080725.htm http://www.faqs.org/sports-science/A-Ba-and-timeline/ACL-Injuries-andFemale-Athletes.html http://www.orthopodsurgeon.com/aclinwomen.html http://sportsmedicine.about.com/od/injuryprevention/a/ACL_prevention.htm http://www.infosports.com/clvclinic/femaleacl.htm http://espn.go.com/trainingroom/s/acl.html http://willtaft.com/842/health/preventing-acl-injuries-in-women http://www.elitefts.com/documents/acl.htm https://www.femaleathletesfirst.com/aclinjuries.aspx?animation=off http://www.aclsolutions.com/theacl_4.php http://www.orthosupersite.com/view.aspx?rid=2093 http://blog.lib.umn.edu/gann0042/femaleaclinjuries/ http://www.docstoc.com/docs/47548399/ACL-injuries-in-female-athletes http://www.ur.umich.edu/0001/Mar05_01/11.htmhttp://www. sportsmetrics.net/?page_id=48 http://www.newsweek.com/2008/01/07/weak-at-the-knees.htmlhttp:// www.ur.umich.edu/0001/Mar05_01/11.htm http://www.nismat.org/ptcor/female_kneehttp://www.newsweek.com/2008/ 01/07/weak-at-the-knees.html http://billingsgazette.com/news/local/article_a314d6b2-52e8-56de-80289273b44d1a79.htmlhttp://www.nismat.org/ptcor/female_knee http://www.bodybuilding.com/fun/peak43.htmhttp://billingsgazette.com/ news/local/article_a314d6b2-52e8-56de-8028-9273b44d1a79.html http://www.nytimes.com/specials/women/warchive/960814_1176.html http://www.bodybuilding.com/fun/peak43.htm http://www.npr.org/templates/story/story.php?storyId=93309486http:// www.nytimes.com/specials/women/warchive/960814_1176.html http://chicagosportsmedicine.com/knee_injuries_women.htmlhttp://www. npr.org/templates/story/story.php?storyId=93309486 http://www.women-running-together.com/kneeinjuries.htmlhttp:// chicagosportsmedicine.com/knee_injuries_women.html http://ash.org/knees.htmlhttp://www.women-running-together.com/ kneeinjuries.html http://www.sportsinjurybulletin.com/archive/ female-knee-injuries.htmlhttp://ash.org/knees.html http://www.arthritistoday.org/news/knee-injuries-female-soccer031.php http://www.sportsinjurybulletin.com/archive/female-knee-injuries.html http://www.pponline.co.uk/encyc/female-athletes-strength-trainingexercises-for-knee-injuries-158http://www.arthritistoday.org/news/kneeinjuries-female-soccer031.php http://www.webmd.com/pain-management/knee-pain/women-avoid-kneeinjury
References [1] Internet world stats: usage and population statistics. Internet World Statistics; December 31, 2011 [Accessed May 1, 2012, bhttp://www.internetworldstats. com/stats.html>]. [2] International telecommunications union data and statistics. International Telecommunications Union; April 4, 2012 [Accessed May 1, 2012, bhttp://www.itu. int/ITU-D/ict/statistics/>]. [3] Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F. Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use? Int J Med Inform 2004;74:13–9. [4] Couper MP, Singer E, Levin CA, Fowler FJ, Fagerlin A, Zikmund-Fischer BJ. Use of the internet and ratings of information sources for medical decisions: results from the DECISIONS survey. Med Decis Making 2010;30:106S. [5] Beredjiklian PK, Bonzentka DJ, Steingberg DR, Bernstein J. Evaluating the source and content of orthopaedic information on the Internet: the case of carpal tunnel syndrome. J Bone Joint Surg Am 2000;82:1540–3. [6] Mathur S, Shanti N, Brkaric M, Sood V, Kubeck J, Paulino C, et al. Surfing for scoliosis: the quality of information available on the internet. Spine 2005;30:2695–700.
202
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
[7] Starman J, Gettys K, Capo J, Fleischli JE, Norton HJ, Karunakar MA. Quality and content of internet-based information for ten common orthopaedic sports medicine diagnoses. J Bone Joint Surg 2010;92:1612–8. [8] Health on the net code of conduct for medical and health websites. Health on the Net Foundation; April 4, 2012 [Accessed May 3, 2012, bhttp://www.hon.ch.HONcode/ Pro/Conduct.html>]. [9] Winker MA, Flanagin A, Chi-Lum B, White J, Andrews K, Kennett RL, et al. Guidelines for medical and health information sites on the Internet. JAMA 2000;283:1600–6. [10] Griffin LY, Albohm MJ, Arendt EA, Bahr R, Beynnon BD, Demaio M, et al. Understanding and preventing noncontact anterior cruciate ligament injuries: a review of the Hunt Valley II meeting, January 2005. Am J Sports Med 2006;34(9):1512–32. [11] Top Ten Search Engines List. List of search engines. Accessed May 1, 2012, http:// www.listofsearchengines.info/; February 1, 2012. [12] comScore: Yahoo's search share drops again in December. Search Engine Watch. Accessed May 1, 2012, http://searchenginewatch.com/article/2048881/comScoreYahoos-Search-Share-Drops-Again-in-December; January 18, 2011.
[13] Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002;324:573–7. [14] Kutner M, Greenberg E, Baer J. National assessment of adult literacy (NAAL). A first look at the literacy of America's adults in the 21st century. National Center for Education Statistics; December 15, 2005 [Accessed June 2, 2011, bhttp:// nces.ed.gov/NAAL/index.asp>]. [15] American Academy of Orthopaedic Surgeons (AAOS) disclosure program. American Academy of Orthopaedic Surgeons; February 2012 [Accessed May 27, 2012, bhttp:// www3.aaos.org/education/disclosure/disclosure_intro.cfm>]. [16] Ley P, Florio T. The use of readability formulas in healthcare. Psychol Health Med 1996;1:7–28. [17] Braddock CH. The emerging importance and relevance of shared decision making to clinical practice. Med Decis Making 2010;30:5S–7S.
Contents lists available at SciVerse ScienceDirect
The Knee
Examining internet resources on gender differences in ACL injuries: What patients are reading Michelle M. Gosselin a,⁎, Mary K. Mulcahey b, Edward Feller c, Michael J. Hulstyn b a b c
Warren Alpert Medical School of Brown University, United States Department of Orthopaedics, Warren Alpert Medical School of Brown University, United States Department of Community Health, Warren Alpert Medical School of Brown University, United States
a r t i c l e
i n f o
Article history: Received 29 June 2012 Received in revised form 31 October 2012 Accepted 10 November 2012 Keywords: Anterior cruciate ligament (ACL) injuries Internet information Gender differences Female ACL
a b s t r a c t Background: The Internet is a popular and powerful resource used by both clinicians and patients. However, medical information on the Internet is often unregulated, biased, and of poor quality. Given the frequency of ACL injuries in young women, this topic was used as a case study to examine the information available to patients on the Internet. Methods: An Internet search was performed using the top three general search engines for four different search phrases related to ACL injuries in females. The top ten sites from each search were analyzed on the basis of both website interface and a specific content scoring guide. Results: A total of 35 unique websites were analyzed. The majority of websites were authored by a layperson or had an unidentified author. Readability data indicated that many sites were written at literacy levels that were too difficult for the average reader. Finally, the average content analysis score for all sites was 41.6 out of a possible 100 points (range 10–87.5); only one-third of the websites cited references for the information provided. Conclusions: We found the Internet information on ACL injuries in women to be largely unsupported, outdated and too difficult for the average reader to comprehend. The average site covered less than half of the topics that we considered relevant to ACL injuries in females indicating that information available on the Internet is largely incomplete. Clinical relevance: Clinicians should be aware of the deficiencies in Internet information in order to appropriately address these issues with patients. © 2012 Elsevier B.V. All rights reserved.
1. Introduction 1.1. Internet use The Internet has become an integral part of our society. In 2011, it was estimated that approximately 80% of the United States population, or more than 270 million people, used the Internet [1]. Further, roughly 2.5 billion people worldwide used the Internet in 2011, indicating that the Internet is rooted in cultures all over the globe [2]. The Internet continues to revolutionize the way people communicate and obtain information. It has been found that searching for health information using an Internet general search engine, such as Google, is the third most common use of the Internet behind email and product research [3]. The DECISIONS study revealed that patients rated information they received from the Internet second only to information they received from their healthcare provider [4]. This suggests that patients not only access the Internet for information regarding their ⁎ Corresponding author at: Box G-9045, 222 Richmond Street, Providence, RI 02903, United States. Tel.: +1 203 640 5521; fax: +1 401 444 3609. E-mail address: [email protected] (M.M. Gosselin). 0968-0160/$ – see front matter © 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.knee.2012.11.006
health, but that they accept what they read as fact. The unregulated nature of the Internet, allows anyone to publish health related information. This reality puts patients at high risk of making decisions about their health based on poor quality information. 1.2. Quality of online materials Several studies have assessed Internet use and the quality of health information available online related to specific conditions such as scoliosis, carpal tunnel syndrome, and a variety of sports medicine diagnoses [5–7]. Generally, the studies revealed poor quality and accuracy of webbased information related to these topics. An evaluation of Internet information regarding carpal tunnel syndrome revealed that 23% of the sites offered unconventional or misleading information [5]. Similarly, a study examining websites related to scoliosis found that 90% of the websites scored less than 16 out of a possible 32 points for content quality and only 3 of 50 sites examined were between 76 and 100% accurate [6]. With increasing awareness of the deficiencies of web-based health information, many organizations have attempted to create specific standards or criteria to evaluate website content and guide patients
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
on how to critically read these sites. The Health on the Net (HON) Foundation and the American Medical Association (AMA) are among the many groups attempting to establish a set of guidelines to ensure that the information read by consumers meets certain standards [8,9]. Websites can receive certification from these organizations indicating that their website upholds the criteria established by that organization. A review of Internet information on ten common orthopedic sports medicine diagnoses found a correlation between presence of HON seal and greater content and website credibility scores [7]. However, these principles go largely unused by both health information consumers and website developers; only 37 of the 154 (24%) websites examined in this study had the HON seal [7], increasing the risk of unregulated, inaccurate and outdated Internet health information. Injury to the anterior cruciate ligament (ACL) is a very common and costly problem, especially among young and otherwise healthy athletes. It is estimated that between 80,000 and 250,000 ACL injuries occur in the United States each year with more than 50% of these occurring in athletes aged 15–25 years old [10]. This age group comprises 45% of Internet users making them especially susceptible to reading information about ACL injuries online [2]. Given the financial and individual costs of this condition, ACL injuries affect many people in the patient's life. We speculate that parents, coaches, athletic trainers and physical therapists likely turn to the Internet for information regarding injuries like an ACL tear as patients often encounter these people prior to seeing an orthopaedic surgeon. Using ACL injuries in women as a case study, we evaluated general trends in the Internet resources that patients, parents, coaches or athletic trainers might access for information regarding this condition. To our knowledge, there is no previous literature on this subject. The goal of this study is to critically assess the quality of the information regarding ACL injuries in women available to patients on the Internet.
2. Materials and methods 2.1. Search methodology An algorithm illustrating our search methods is shown in Fig. 1. We performed a search using selected keywords on the top three general search engines: Google, Yahoo and Bing [11]. The Yahoo search engine was powered by Bing, so effectively there were two separate sets of results for each search term since the Yahoo and Bing searches yielded the same set of websites. Data from December 2011 revealed that Google was used for 66.6% of all general Internet searches in the US, followed by 16% on Yahoo and 12% on Bing [12], indicating that we examined 95% of the general searches the average individual performed. We chose to use general search engines, as opposed to a medical information specific search, since patients more often turn to these general searches when looking for health information. To control for the variation in keywords searched by patients, we used four separate, but similar, search terms: “ACL injuries in women”, “ACL injuries in female athletes”, “knee injuries in women”, and “knee injuries in female athletes”. Searches performed using the more general phrases “ACL injuries” or “knee injuries” yielded a large proportion of news articles regarding professional athletes who
197
had recently suffered knee injuries. We did not believe this would be a fruitful search topic and thus narrowed the search terms to address the gender differences in ACL/knee injuries. All searches were performed on February 10, 2012 to account for the day-to-day variation in the top ten sites returned with each search query. Data regarding number of hits for each search term on the three different search engines is reported in Table 1. Most people do not look beyond the first page of search results that lists the top websites for a given search [13]thus we chose to evaluate only the top ten search results for each query. The websites were sorted; all duplicates and non-functioning website links were removed to compile a master list of unique websites to be analyzed (Appendix A). We did not exclude any websites on the basis of content or design, as we sought to evaluate the webpage that any given patient may read after performing a general search. 2.2. Data evaluation The website interface was assessed for general information regarding the site design, authorship and readability. Website content was also evaluated on the basis of completeness, validity, and supporting references specific to gender differences in ACL injuries. 2.3. Authorship The websites were first classified into one of seven categories: physician, non-physician, academic, commercial product, commercial site, layperson or unidentified. We used the following definitions, similar to that previously described [5]: physician: author or authors were individual or group physicians with no research or university group, non-physician: all alternative medical providers including chiropractors, physical and occupational therapists, academic: author or authors were affiliated with a university or research group, commercial site: the author represented a commercial web site without an interest in a specific commercial product, commercial product: an author or authors who were marketing a specific commercial product related to ACL injuries or prevention, layperson: individuals or organizations who did not belong to any of the previous categories and who maintained a noncommercial web site for providing information about knee injury, or unidentified: the author, or author's credentials, were not specified. Two examiners categorized each site independently; in cases of disagreement, a third examiner cast the deciding vote. We also sought to determine whether a third-party group (e.g. Health on the Net [HON] Foundation or the American Medical Association [AMA]) certified the websites as indicated by the presence of the entity's seal. 2.4. Readability analysis We assessed the readability of the website text using the Flesch– Kincaid grade and the Flesch reading ease tools [3,14]. This assessment was performed using the readability statistics function in Microsoft Word 2007. The health information readability goals, Flesch–Kincaid
Fig. 1. Algorithm for the search methodology. This algorithm illustrates the step by step website search process to show how we arrived at our final database of websites for evaluation.
198
M.M. Gosselin et al. / The Knee 20 (2013) 196–202 Table 1 This table lists the number of hits obtained with each search query on each of the general search engines used. Search query “ACL injuries in women” Google Yahoo Bing “ACL injuries in female athletes” Google Yahoo Bing “Knee injuries in women” Google Yahoo Bing “Knee injuries in female athletes” Google Yahoo Bing
# of hits 164,000 909,000 847,000
35,700 194,000 183,000
3,050,000 14,400,000 19,600,000
104,000 3,230,000 3,220,000
grade goal of b8 and the Flesch reading ease goal of > 60, have been set based on current literacy levels in the United States [14].
Table 3 Website content parameters. This table outlines the content scoring guide used to evaluate each website's content. The specific topics we examined are listed in the left column and the number of points allotted to each topic is noted in the right column; a total of 100 points were possible. Specific topic Knee anatomy Gender differences Gender difference statistics Hormonal differences Narrow intercondylar notch Knee angle Decreased strength (quadriceps/hamstring ratio) Neuromuscular factors Mechanism of injury Contact Non-contact Diagnosis: physical exam, imaging Treatment options Operative treatment Different graft options Non-operative treatment Treatment complications/long-term sequelae Addresses Prevention Strategies Braces Pre-competition Total
Points 5 5 5 5 5 5 5 5 5 10 5 5 5 15 5 5 5 100
3. Results
2.5. Website interface criteria Each individual site was further evaluated on the basis of the criteria listed in Table 2, with the goal of assessing the set-up, accessibility, and currency of the information contained on the website. These criteria were developed using the concepts outlined by the HON guidelines.
2.6. Website content analysis Content analysis of each website was performed using a point system developed by the authors (Table 3). Each site was evaluated for the presence of specific topics relevant to gender differences in ACL injuries including knee anatomy, potential reasons for female susceptibility to ACL injuries, diagnosis, treatment, and prevention strategies. The website was given the specified number of points if that topic was covered within the text of the site. Each website was evaluated and scored separately by two independent examiners. The content score was calculated by averaging the scores of two independent examiners and the inter-observer variability was noted. We also assessed whether each topic was supported by outside references and analyzed the number, quality, and currency of references provided.
After excluding all duplicate and non-functioning links, a list of 35 unique websites was compiled for analysis (Appendix A). 3.1. Website interface evaluation Only 18 of 35 websites (51%) provided a date of creation or date of most recent update; of those 18 sites, only three were updated within the year prior to the search. Sixteen websites (46%) had advertisements present and 12 websites (35%) had evidence of sponsorship, while 12 websites (35%) had evidence of a commercial intent, all of which had links to said commercial interest. These sites were often physician or non-physician authored sites with links to their practice, hospital or clinic, without explicit marketing intent. Only two websites (6%) had a third party seal (e.g. HON, AMA, etc.) to signify the certification by one of the organizations established to uphold a standard of health information on the Internet (Fig. 2). 3.2. Authorship The percentages of each category of authorship are shown in Fig. 3. The majority of the websites were authored by a layperson (37%) or the author was unidentified (23%). Ten websites (28%) were either authored by a physician or someone affiliated with an academic institution (i.e. a researcher or other healthcare provider). Nine of the
Table 2 Website interface criteria. This table lists the criteria used to evaluate each website on the basis of website layout and general set-up. Date of last update present Copyright Notice Statement of purpose Disclosure of authorship Author credentials Financial disclosure Presence of ads Presence of sponsorship Privacy statement Subscription/log in Contact information Commercial interest Links to commercial interest Third party seal (i.e. HON, AMA, etc.)
Fig. 2. Website interface characteristics. This graph illustrates the number of websites that exhibited each of the search parameters.
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
199
Table 4 Content analysis by website category. This table shows the average content score arranged by website category. Category
# of websites
(0–100)
Flesch–Kincaid grade (goal b8)
Flesch reading ease (goal >60)
Academic Physician Non-physician Commercial product Lay Unidentified Total
3 7 2 2 13 8 35
55.0 49.3 53.8 32.5 37.9 35.3 41.6
12.7 10.7 12.3 12.9 11.5 12.3 11.7
37.8 52.0 44.5 38.5 49.3 45.9 47.4
Fig. 3. Categorization of websites. This graph shows the percentage of websites that were classified in each category. The majority (60% of the total number of websites assessed) were classified as either lay person or the author was unidentified. Conversely, less than one-third (28% of the total number of websites) were authored by a physician or someone affiliated with an academic institution. Note: 0 websites were classified as “Commercial Site” so this category was omitted from the graph.
met the Flesch–Kincaid Grade goal and five met the Flesch Readability Score goal). Further, there was no correlation between readability scores and content scores of the websites we examined. Table 4 presents the average readability statistics broken down by website category.
websites (26%) were articles from a newspaper or other news outlet (i.e. television, radio) and three of the websites (9%) were blogs. Only two sites (6%) were classified as selling a specific commercial product, while no site was classified as a commercial site explicitly. This finding was different from results published on a similar search on “carpal tunnel syndrome”, which revealed that one-third of the websites endorsed a commercial product [5].
The content analysis results are presented in Fig. 5. The informational content scores were calculated for each website by two independent evaluators and the scores were averaged to give a single content score for each site. The mean informational content score for all sites was 41.6 out of a possible 100 points (range: 10–87.5). The mean informational content score broken down by website category is presented in Table 4. The academic, non-physician and physician classified sites had higher average content scores (55.0, 53.8 and 49.3 respectively) compared to the layperson, unidentified and commercial product sites (37.9, 35.3 and 32.5 respectively). Further, the intra-observer reliability of the informational content score was assessed. The mean difference in scores between the two observers was 1.4 points. Eighty-five percent of the websites were scored within five points of each other. Diagnosis, treatment and long-term sequelae were discussed on less than half the websites, most of which were physician or academic in classification. Gender difference statistics, risk factors and prevention strategies were discussed on 80% of the sites. Anatomy was well described by all site classifications except for layperson authored websites. Interestingly, only 12 of the 35 websites provided references for the information that was presented (Fig. 6). All academic (3), non-physician (2) and commercial product (2) classified sites provided references for the information presented on their websites.
3.3. Readability analysis The readability scores for 34 sites are shown in Fig. 4. One website was in a PDF format and could not be copied into Word 2007 to complete the readability statistics assessment. The mean Flesch–Kincaid Grade (goal b8) for all 34 websites was 11.77 ± 2.13 and the mean Flesch Readability Score (goal >60) was 47.4 ± 10.87; both scores are significantly above goal reading levels for health information for the general population based on previous studies [3,14]. Ninety-one percent of websites were too difficult according to the Flesch–Kincaid Grade goal of b8 and 86% were above the goal reading level according to Flesch Readability Score goal of >60 (only three websites
3.4. Content analysis
Fig. 4. Website readability. This graph depicts the Flesch–Kincaid grade and the Flesch reading ease scores of each website analyzed. The red lines indicate the goal reading levels of health materials (Flesch–Kincaid grade (goal of b8) and the Flesch reading ease (goal of >60)) given the literacy of US adults. Less than 5 websites achieved these goals indicating that, overall, the websites were too difficult for the average US adult to fully comprehend.
200
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
Fig. 5. Website content analysis scores. This graph illustrates the content analysis scores obtained by two independent reviewers, broken down by website. Both scorers scored websites consistently throughout with an average difference between the two reviewers of 1.4 points. The final column shows the average content score of all websites examined for the two reviewers.
Among the 12 websites that provided references, many of those references were outdated, incomplete or contained non-functional links. The three academic sites provided the greatest number and most up-to-date and complete set of references of these 12 sites; otherwise, there was no correlation between presence of references and overall content score.
4. Discussion Several studies have examined Internet information regarding various orthopedic conditions [5–7]. Many of these studies have found that information online is outdated, unsubstantiated, and often has a commercial interest present. Further, content analyses have shown that the information is largely incomplete and in some cases, inaccurate [5–7]. There is no standardized way of regulating the type or quality of information available on the Internet, so physicians need to be aware of the potential problem of misinformation available online. Our findings concur with literature available regarding medical information on the Internet. In our study, 37% of the websites identified were authored by a layperson and 23% had unidentified authors, indicating that the majority of sites the health information consumer reads are either written by someone outside the medical profession or the author is not disclosed. Yet, health consumers generally believe what they read on the Internet without questioning the source of the information. Our study revealed that many websites were outdated, only 3 with evidence of having been updated within the last year.
Fig. 6. References provided by each website. This graph shows the number of websites that provided a given number of references for the information presented on their site. As can be noted, the majority of the websites did not provide any references while very few provided more than 5 references.
Given that 12 of 35 websites analyzed showed some form of commercial interest, the intent and motives behind the information supplied can be called into question. It is possible that the website sponsor or author is providing biased information as a means of promoting a specific product, practice or service. The lack of current, high-quality references also questions the validity and accuracy of information on many sites. Most websites did not cite specific references and on those websites where references were provided, many were outdated and of poor quality. Unlike physician disclosure requirements, these websites are not required to disclose any financial or commercial relationships, as the sites are largely unregulated. Most institutions, societies and journals require physicians and researchers to disclose any commercial relationship or potential conflict of interest. One of the goals of the Disclosure Program of the AAOS is to: increase transparency throughout AAOS by making this disclosure program available to AAOS members and the public [15]. The AMA and HON have attempted to improve the transparency and accountability of these websites through their established guidelines, but studies have indicated that these programs are underutilized. The HON website notes that 7300 websites are registered and certified through their program, while there are over 50,000 websites containing medical information [8], implying that less than 15% of websites are HON certified. The fact that only two websites (6%) in our study had certification by a third party raises several concerns. First, it is possible that website designers are not aware of the possibility of obtaining a “seal of approval” to indicate that the site has met certain accepted standards. Second, it could illustrate that the websites accessed by health consumers do not uphold the credible guiding principles of these organizations. Regardless, there should be greater standardization of certification modalities. Numerous organizations have established guidelines for health information on the Internet; information regarding the third party seal could be more efficiently distributed if there was only one organization governing the standards. Further, it would be easier to educate patients regarding one set of principles to help them more effectively, and critically, read health information on the Internet. Our results raise another concern not addressed in similar studies: the readability of most of the online health information we examined was far too difficult for the average American consumer. It has been reported that ninety-two million Americans have less than eighthgrade level reading skills [14], yet the average reading level of the websites we examined was above the eleventh-grade level. Less than ten percent of the websites met both the Flesch–Kincaid Grade
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
and the Flesch Readability Score goal levels. This indicates that many patients, parents, coaches or trainers may not be able to fully comprehend the information read while performing an online search. As the field of medicine continues to move toward a shared decision making model, patient's comprehension of medical information is becoming increasingly important. It has been shown that increasing readability leads to improvement in understanding and often also improved cooperation with treatment [16]. Further, it is thought to be less important that patients have complete knowledge recall regarding a particular topic, but that they conceptually understand their disease, injury or treatment plan so that they may play an active role in decision making [17]. Improving readability of patient information has the potential to improve patient satisfaction and ultimately patient care. Finally, we analyzed the topics discussed on the websites accessed with simple general search engine queries. Since we focused our search terms on women/females, it was not surprising that our searches turned up articles focused on gender differences in ACL injuries as well as possible prevention strategies. All but two websites discussed at least one of the risk factors associated with an increased risk of knee injury in women. We found that many websites lacked information on the signs of a serious injury, when to see a doctor, possible treatment options as well as long-term problems that could arise from this injury. Our study has several limitations. We used only the top three general search engines and did not include any medical specific search engines (i.e. Medline, WebMD). These three search engines, however, comprised 95% of the general searches performed by laypeople [12]. Further, we examined only the top ten websites from each search query, which gave us a limited number of websites to evaluate. However, given previous data regarding consumer Internet habits, we felt that this search strategy was most similar to that used by patients when researching health information online and gave us a total number of websites similar to previous studies. Additionally, we used a very specific search topic. It is possible that other orthopedic topics are covered in more detail on the Internet. We limited our search to the English language; searches using other languages may yield different results. Finally, our content analysis simply acknowledged the presence of specific information regarding a myriad of topics related to ACL/knee injuries in women. We attempted to make our grading scale standardized and objective, however, the possibility of observer bias in examiners' website grading cannot be excluded. In conclusion, we found the Internet information on ACL injuries in women to be largely unsupported, outdated and too difficult for the average reader to comprehend. The average site covered less than half of the topics that we considered relevant to ACL injuries in females indicating that information available on the Internet is largely incomplete. Internet website authors should focus on readability of patient information, as this has the potential to improve patient satisfaction and ultimately patient care. Future studies should be aimed at developing a set of strategies that consumers could employ to more effectively and critically read information they access on the Internet. Conflict of interest statement None of the authors have any financial or personal relationship with other people or organizations that may bias this study. Acknowledgments All authors made significant contributions to the development of the study design, completion of study, as well as preparation and editing of the manuscript; there were no other contributors to the study. This study has no source or funding and thus has no study sponsor involvement.
201
Appendix A. Listing of websites evaluated
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
35
http://www.youcanbefit.com/ACL.html http://articles.cnn.com/2007-03-26/health/hm.acl.girls_1_acl-injuriesfemale-athletes-knee-braces?_s=PM:HEALTH http://www.hughston.com/hha/a_11_3_2.htm http://www.thesportjournal.org/article/prevention-acl-injuries-femaleathletes-through-early-intervention http://orthopedics.about.com/od/aclinjury/tp/acl.htm http://media.ncaa.org/Skins/ACLSkin/Default.aspx?PageId=6c28f11c-85a74a22-911d-41e934e98e6b&ss=MediaPortal http://www.momsteam.com/health-safety/muscles-joints-bones/knee/aclinjuries-in-female-athletes http://bjsm.bmj.com/content/42/6/394.full http://www.cdc.gov/media/pressrel/2008/r080725.htm http://www.faqs.org/sports-science/A-Ba-and-timeline/ACL-Injuries-andFemale-Athletes.html http://www.orthopodsurgeon.com/aclinwomen.html http://sportsmedicine.about.com/od/injuryprevention/a/ACL_prevention.htm http://www.infosports.com/clvclinic/femaleacl.htm http://espn.go.com/trainingroom/s/acl.html http://willtaft.com/842/health/preventing-acl-injuries-in-women http://www.elitefts.com/documents/acl.htm https://www.femaleathletesfirst.com/aclinjuries.aspx?animation=off http://www.aclsolutions.com/theacl_4.php http://www.orthosupersite.com/view.aspx?rid=2093 http://blog.lib.umn.edu/gann0042/femaleaclinjuries/ http://www.docstoc.com/docs/47548399/ACL-injuries-in-female-athletes http://www.ur.umich.edu/0001/Mar05_01/11.htmhttp://www. sportsmetrics.net/?page_id=48 http://www.newsweek.com/2008/01/07/weak-at-the-knees.htmlhttp:// www.ur.umich.edu/0001/Mar05_01/11.htm http://www.nismat.org/ptcor/female_kneehttp://www.newsweek.com/2008/ 01/07/weak-at-the-knees.html http://billingsgazette.com/news/local/article_a314d6b2-52e8-56de-80289273b44d1a79.htmlhttp://www.nismat.org/ptcor/female_knee http://www.bodybuilding.com/fun/peak43.htmhttp://billingsgazette.com/ news/local/article_a314d6b2-52e8-56de-8028-9273b44d1a79.html http://www.nytimes.com/specials/women/warchive/960814_1176.html http://www.bodybuilding.com/fun/peak43.htm http://www.npr.org/templates/story/story.php?storyId=93309486http:// www.nytimes.com/specials/women/warchive/960814_1176.html http://chicagosportsmedicine.com/knee_injuries_women.htmlhttp://www. npr.org/templates/story/story.php?storyId=93309486 http://www.women-running-together.com/kneeinjuries.htmlhttp:// chicagosportsmedicine.com/knee_injuries_women.html http://ash.org/knees.htmlhttp://www.women-running-together.com/ kneeinjuries.html http://www.sportsinjurybulletin.com/archive/ female-knee-injuries.htmlhttp://ash.org/knees.html http://www.arthritistoday.org/news/knee-injuries-female-soccer031.php http://www.sportsinjurybulletin.com/archive/female-knee-injuries.html http://www.pponline.co.uk/encyc/female-athletes-strength-trainingexercises-for-knee-injuries-158http://www.arthritistoday.org/news/kneeinjuries-female-soccer031.php http://www.webmd.com/pain-management/knee-pain/women-avoid-kneeinjury
References [1] Internet world stats: usage and population statistics. Internet World Statistics; December 31, 2011 [Accessed May 1, 2012, bhttp://www.internetworldstats. com/stats.html>]. [2] International telecommunications union data and statistics. International Telecommunications Union; April 4, 2012 [Accessed May 1, 2012, bhttp://www.itu. int/ITU-D/ict/statistics/>]. [3] Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F. Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use? Int J Med Inform 2004;74:13–9. [4] Couper MP, Singer E, Levin CA, Fowler FJ, Fagerlin A, Zikmund-Fischer BJ. Use of the internet and ratings of information sources for medical decisions: results from the DECISIONS survey. Med Decis Making 2010;30:106S. [5] Beredjiklian PK, Bonzentka DJ, Steingberg DR, Bernstein J. Evaluating the source and content of orthopaedic information on the Internet: the case of carpal tunnel syndrome. J Bone Joint Surg Am 2000;82:1540–3. [6] Mathur S, Shanti N, Brkaric M, Sood V, Kubeck J, Paulino C, et al. Surfing for scoliosis: the quality of information available on the internet. Spine 2005;30:2695–700.
202
M.M. Gosselin et al. / The Knee 20 (2013) 196–202
[7] Starman J, Gettys K, Capo J, Fleischli JE, Norton HJ, Karunakar MA. Quality and content of internet-based information for ten common orthopaedic sports medicine diagnoses. J Bone Joint Surg 2010;92:1612–8. [8] Health on the net code of conduct for medical and health websites. Health on the Net Foundation; April 4, 2012 [Accessed May 3, 2012, bhttp://www.hon.ch.HONcode/ Pro/Conduct.html>]. [9] Winker MA, Flanagin A, Chi-Lum B, White J, Andrews K, Kennett RL, et al. Guidelines for medical and health information sites on the Internet. JAMA 2000;283:1600–6. [10] Griffin LY, Albohm MJ, Arendt EA, Bahr R, Beynnon BD, Demaio M, et al. Understanding and preventing noncontact anterior cruciate ligament injuries: a review of the Hunt Valley II meeting, January 2005. Am J Sports Med 2006;34(9):1512–32. [11] Top Ten Search Engines List. List of search engines. Accessed May 1, 2012, http:// www.listofsearchengines.info/; February 1, 2012. [12] comScore: Yahoo's search share drops again in December. Search Engine Watch. Accessed May 1, 2012, http://searchenginewatch.com/article/2048881/comScoreYahoos-Search-Share-Drops-Again-in-December; January 18, 2011.
[13] Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002;324:573–7. [14] Kutner M, Greenberg E, Baer J. National assessment of adult literacy (NAAL). A first look at the literacy of America's adults in the 21st century. National Center for Education Statistics; December 15, 2005 [Accessed June 2, 2011, bhttp:// nces.ed.gov/NAAL/index.asp>]. [15] American Academy of Orthopaedic Surgeons (AAOS) disclosure program. American Academy of Orthopaedic Surgeons; February 2012 [Accessed May 27, 2012, bhttp:// www3.aaos.org/education/disclosure/disclosure_intro.cfm>]. [16] Ley P, Florio T. The use of readability formulas in healthcare. Psychol Health Med 1996;1:7–28. [17] Braddock CH. The emerging importance and relevance of shared decision making to clinical practice. Med Decis Making 2010;30:5S–7S.