- Email: [email protected]
Assessment of the Quality and Content of Information on Anterior Cruciate Ligament Reconstruction on the Internet
Current Concepts
Assessment of the Quality and Content of Information on Anterior Cruciate Ligament Reconstruction on the Internet Robert A. Bruce-Brand, M.R.C.S.I., Joseph F. Baker, M.R.C.S.I., Damien P. Byrne, Ph.D., Niall A. Hogan, F.R.C.S.I., and Tom McCarthy, F.R.C.S.I.
Abstract: The Internet has become a major source of health information for the public. However, there are concerns regarding the quality, accuracy, and currency of medical information available online. We assessed the quality of information about anterior cruciate ligament (ACL) reconstruction on the first 60 websites returned by the 4 most popular search engines. Each site was categorized by type and assessed for quality and validity using the DISCERN score, the Journal of the American Medical Association (JAMA) benchmark criteria, and a novel ACL reconstructionespecific content score. The presence of the Health On the Net Code (HONcode), a purported quality assurance marker, was noted. The quality of information on ACL reconstruction available online is variable, with many websites omitting basic information regarding treatment options, risks, and prognosis. Commercial websites predominate. Academic and allied health professional websites attained the highest DISCERN and JAMA benchmark scores, whereas physician sites achieved the highest content scores. Sites that bore the HONcode seal obtained higher DISCERN and ACL reconstruction content scores than those without this certification. The HONcode seal is a reliable indicator of website quality, and we can confidently advise our patients to search for this marker.
P
hysicians should be cognizant of the information their patients are likely to be accessing online and be able to steer them to high-quality websites. The Internet’s exponential growth has triggered an information revolution, and it has evolved into a major source of health information for the public. More than 273 million Americans, representing 78.6% of the population, now have Internet access.1 More than half use the World Wide Web to obtain health or medical information, and the majority of them seek health information online at least once a month.2 However, although the Internet represents an unprecedented method of sharing and accessing knowledge, there are growing concerns about the quality and currency of the medical information available. Several studies have
From the Department of Orthopaedic Surgery, St. James’s Hospital (R.A.B-B, J.F.B., N.A.H., T.M.); and Department of Orthopaedic Research, Orthopaedic Research & Innovation Foundation (D.P.B), Dublin, Ireland. The authors report that they have no conflicts of interest in the authorship and publication of this article. Received November 6, 2012; accepted February 1, 2013. Address correspondence to Robert A. Bruce-Brand, M.R.C.S.I., St James’s Hospital, James’s St, Dublin 8, Ireland. E-mail: [email protected] Ó 2013 by the Arthroscopy Association of North America 0749-8063/12726/$36.00 http://dx.doi.org/10.1016/j.arthro.2013.02.007
shown that many health sites contain misleading, inaccurate, and inappropriate information.3-12 Injury of the anterior cruciate ligament (ACL) is common in athletes, with an annual incidence of about 35 per 100,000 of the general population.13 It is one of the most commonly treated conditions of the knee.14 Estimated rates of ACL reconstruction exceed 175,000 per year in the United States.15 The Internet is an important source for patients seeking information about ACL reconstruction surgery. Cailliez et al.16 investigated patient knowledge ahead of ACL reconstruction and found that 40% of patients had researched on the Internet for surgery-related information. Koh et al.17 examined factors affecting patients’ graft choice in ACL reconstruction and found that the Internet was the most important contributor in 21.7% of cases. However, we are not aware of any study that has determined the quality of information about ACL reconstruction found on the Internet. The aim of this study, therefore, was to assess the quality of information about ACL reconstruction on the Internet using recognized scoring systems, identification of quality markers, and a novel specific score. Scoring systems included the Journal of the American Medical Association (JAMA) benchmark and DISCERN criteria.11,18 Initiatives such as the Health On the Net code (HONcode) identify websites that agree to comply with a code
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol 29, No 6 (June), 2013: pp 1095-1100
1095
1096
R. A. BRUCE-BRAND ET AL.
of ethics to provide quality objective and transparent medical information.19
Searching on the Web We used the search term ACL reconstruction on the 4 most popular search engines: Google, Yahoo!, Bing, and Ask. Google is the dominant search engine, composing 83.06% of the total market share, followed by Yahoo! (6.86%), Bing (4.27%), and Ask (0.51%) (figures as of August, 2012).20 We had planned to review the first 60 websites encountered during an Internet search. To reflect the disproportionate use of the various available search engines, we analyzed the first 30 of approximately 985,000 websites returned by Google (www .google.com) and 10 each from Yahoo! (www.search .yahoo.com), Bing (www.bing.com), and Ask (www .ask.com). The total number of hits returned by the search is shown in Table 1. Searches were performed on August 10, 2012. Duplicate websites and those that were inaccessible were excluded from analysis (Fig 1). The first part of the analysis involved determining the type of website. We categorized sites as academic, physician, nonphysician (allied health professionals such as physical therapists, occupational therapists, and alternative medical providers), commercial, media, discussion groups/social networks, and unspecified. These classifications were based on those described in previous work, with the addition of the discussion groups/social media class to reflect the growing trend in sites such as YouTube (www.youtube.com), Facebook (www.facebook.com), and Twitter (www.twitter.com).10,12 Academic websites included those affiliated with a university or medical society. Websites were considered commercial if they displayed advertisements or included products for sale. Physician websites included professional sites for individual physicians as well as practitioner groups not affiliated with an academic institution. Media websites were nonmedical news-oriented sites. Any website that did not fit the preceding categories was classed as unspecified. Websites were then assessed for quality and validity using the DISCERN score, the JAMA benchmark criteria, and an ACL reconstructionespecific content score. They were also assessed for the presence or absence of the HONcode certification.11,18,19 DISCERN, developed by an expert group in the United Kingdom, is an instrument designed to judge the quality of written consumer health information about treatment choices. It determines the quality of a publication Table 1. Search Results for ACL Reconstruction Search Engine Google Yahoo! Bing Ask
Approximate Number of Search Results 985,000 6,450,000 1,710,000 Not available
Fig 1. Internet search flow diagram.
based on 16 questions. The first 8 questions concern the reliability of the publication, and the subsequent 7 questions address specific details of the information about treatment choices. The last question is an overall quality rating of the website. Silberg et al.11 described 4 core criteria to allow readers to determine whether what they were reading was credible, reasonable, or useful. Also known as the JAMA benchmark criteria, they assess authorship, attribution, disclosure, and currency. The authorship criterion requires that the website provide details of authors and contributors, their affiliations, and relevant credentials. Attribution entails listing references and sources for all content and indicating any relevant copyright information. Currency refers to the provision of the dates that content was posted and updated, and disclosure requires that website ownership be prominently and fully disclosed, along with any sponsorship, advertising policies, and potential conflicts of interest. To assess the informational value of each website, we developed an ACL reconstructionespecific content score comprising 25 factors (Table 2). This was done in consultation with sports surgeryetrained consultants and by referencing current peer-reviewed literature.21,22 One point was allocated for the mention of each of the predefined terms that related to general characteristics of the condition, its prognosis, treatment options, and complications. Sites were then scored from 0 to 25, with 25 indicating a site of maximal quality. The Health On the Net Foundation is a nonprofit nongovernmental organization that was established in 1996 and established ethical standards for publishing
1097
ACL RECONSTRUCTION ON THE INTERNET Table 2. ACL ReconstructioneSpecific Content Score General Aspects
Management
Complications
Age Sex Mechanisms Functional disability
Observation Infection Physical therapy Graft failure Bracing Knee pain Reconstruction: Degenerative knee hamstring Functional Reconstruction: requirements patellar tendon Meniscal injury Reconstruction: allograft Timing of surgery Single bundle Anesthesia Double bundle Rehabilitation Anatomic reconstruction Return to sports/work Meniscal repair Revision reconstruction
NOTE. One mark was allocated for each point mentioned on the website.
medical and health-related information on the Internet. The HONcode seal accredits websites that agree to comply with listed standards and publish transparent healthrelated information. For each website, we checked for the presence and validity of the HONcode seal. Statistical analysis was performed using SPSS Statistics, version 17 (SPSS, Chicago, IL). The level of significance was set at 5%. A one-way analysis of variance was used to determine whether website type predicted higher DISCERN, JAMA benchmark, and ACL reconstruction content scores. Tukey multiple comparison methods were used to compare the different means. To determine whether HONcode certification predicted higher scores, 2 sample t tests were used when data was normally distributed, whereas Mann-Whitney U tests were used when data was not normally distributed.
Results A total of 45 unique websites was analyzed (Table 3). Fig 2 shows the breakdown according to website type. The majority were commercial (n ¼ 29 [64%]). Eight (17.8%) of the websites were HONcode certified. The mean overall scores by website type are illustrated in Fig 3. DISCERN Scores The mean DISCERN score for all websites was 41.1 13.3 of a maximum of score of 80 (range, 15 to 78). The highest-scoring website was an academic site. Nineteen websites (42.2%) were scored as being of poor quality, with serious or extensive shortcomings. Only 4 sites attained scores greater than 60, representing excellent quality with minimal shortcomings. The academic and nonphysician website types had the highest overall DISCERN scores (Table 4). However, differences in DISCERN scores were not significantly different between the website types (P > .05). Websites that were HONcode certified had significantly higher mean DISCERN scores (58.0 v 39.8; P ¼ .016).
JAMA Benchmark Criteria The mean JAMA benchmark criteria score was 2.1 1.2. Ten sites had maximal scores. There was a significant difference between the groups (P ¼ .001), with the academic and nonphysician (allied health professionals) sites scoring highest. ACL ReconstructioneSpecific Content Score The highest content score was obtained by an academic site with a score of 24 of 25. One website, of the discussion group type, received 0 points. The mean overall score was 12.29 5.49. The physician-type websites had the highest overall content score, with a mean of 14.4. However, intergroup differences were not significant. HONcode Certification The association between outcome scores and HONcode certification is shown in Fig 4. Websites that bore the HONcode seal obtained higher DISCERN and ACL reconstruction content scores than those without this certification (Table 4). However, JAMA benchmark criteria were not significantly different (2.0 v 2.0; P ¼ .14).
Discussion The Internet has revolutionized the dissemination of information, and patients can access boundless information on any subject, including health.2,12,23 However, unlike traditional media, it allows virtually unregulated publication of information that is available on the widest possible scale. The challenge to the Internet “surfer” therefore is to find credible reliable information among the unregulated mountains of hypertext. The aim of this study was to determine the quality of Internet-based information on ACL reconstruction using recognized quality control measures. Our findings indicate that the websites likely to be accessed by patients are of variable quality. The mean DISCERN score we recorded represents information of only moderate quality, with potentially important but not serious shortcomings. This has the potential to detrimentally alter the patient-physician consultation as well as cause undue anxiety for the patient. The term cyberchondria has been proposed to represent the possible mental state after reading inaccurate information based on the Internet.24 Consistent with previous studies, we found that academic institutionecreated sites provided better information quality.10 However, we also noted that nonphysician (allied health professional) sites achieved the highest mean DISCERN score; this is an inconsistency with previous comparable studies but may result from the small number of nonphysician sites encountered in this study.10 Patients searching the Internet can reliably be directed to look for sites provided by academic institutions.
1098
R. A. BRUCE-BRAND ET AL.
Table 3. List of Unique Websites Evaluated Search Engine Google
Yahoo!
Bing
Ask
Website 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45.
http://en.wikipedia.org/wiki/Anterior_cruciate_ligament_reconstruction http://www.youtube.com/watch?v¼q96M0jRqn7k http://www.nlm.nih.gov/medlineplus/ency/article/007208.htm http://www.hipandkneesurgery.ie/ligament-reconstruction.html http://orthoinfo.aaos.org/topic.cfm?topic¼A00297 http://www.orthogate.org/patient-education/knee/anterior-cruciate-ligament-injuries.html http://orthopedics.about.com/od/aclinjury/tp/acl.htm http://www.webmd.com/a-to-z-guides/anterior-cruciate-ligament-acl-surgery http://www.macdarabutler.com/acl-reconstruction-patella-vs-hamstring-graft/ http://www.aclreconstruction.org/ http://www.mayoclinic.com/health/acl-reconstruction/MY00941/ http://www.scoi.com/aclrecon.htm http://orthopedics.about.com/cs/aclrepain/a/acldecision.htm http://www.orthosports.com.au/content_common/pg-acl-reconstruction.seo http://www.sportsmedalabama.com/patient-education/knee-mcl-acl-reconstruction.html http://drmillett.com/acl-repair-anterior-cruciate-ligament http://www.biomet.com/patients/sportsMedicine/aclReconstruction.cfm http://www.sportmed.ucalgary.ca/ACL http://www.hopkinsmedicine.org/healthlibrary/test_procedures/orthopaedic/knee_ligament_repair_92,P07675/ http://myaclreconstruction.blogspot.com/ http://www.hipsknees.info/flash5/acl_patellor_final.pdf http://www.sharonregional.com/acl-reconstruction.htm http://www.hopkinsortho.org/acl.html http://www.fixknee.com/ACLRecon/ACL%20Recontruction2011.html http://www.aaos.org/news/aaosnow/apr12/cover1.asp http://www.aucklandorthopaedicpractice.co.nz/ligament-reconstruction.html http://www.pitt.edu/wmechbio/publications/graft.pdf http://www.iskinstitute.com/pdf/pdf_forms/ACL%20Reconstruction%20Book.pdf http://www.orlive.com/broward-health/videos/anterior-cruciate-ligament-acl-reconstruction-surgery http://isgweb.aorn.org/ISGWeb/downloads/CEA11002-4055.pdf http://www.bmihealthcare.co.uk/treatment/treatmentsdetail?p_name¼ACL%20reconstruction%20 surgery&p_treatment_id¼307 http://ehealthmd.com/acl-tears/recovery-after-acl-reconstruction http://www.arthroscopy.com/sp05025.htm http://footballsfuture.com/phpBB2/viewtopic.php?p¼14564779 http://www.pamf.org/sports/king/anterior_cruciate.html http://www.kneeclinic.info/problems_acl.php http://www.uwmsk.org/residentprojects/aclreconstruction.html http://www.knee1.com/Education_Center/Therapies/ACL_Reconstruction http://ask.healthline.com/health/acl-reconstruction http://www.ehow.com/facts_5610921_acl-reconstruction.html http://www.anteriorcruciateligament.co.uk/ http://www.londonkneeclinic.com/knee-problems/ligament-injuries#ACL http://www.noc.nhs.uk/hipandknee/information/documents/acl-reconstruction.pdf http://www.oxfordradcliffe.nhs.uk/forpatients/090427patientinfoleaflets/100419acl.pdf http://www.bupa.co.uk/individuals/health-information/directory/a/acl
Our study indicates that for Internet searches on ACL reconstruction, the HONcode certification reliably identifies websites with significantly higher quality and content scores. This mirrors the findings of similar studies investigating the quality of other health care topics.6,12,25 This is reassuring, and as physicians we can direct our patients to look for this quality assurance marker when they search the Internet. Starman et al.12 found that the quality and content of Internet-based information for 10 common orthopaedic sports medicine diagnoses were highly variable, consistent with our findings, with commercial sites dominating search results.12 With the majority of search engine results in this study being commercial sites, there is clear
potential for bias in the information presented. It is necessary for physicians to be aware that their patients may be misinformed and to spend time correcting patients’ perceptions. Starman et al.12 included ACL tear as one of the 10 common orthopaedic sports medicine diagnoses they analyzed on the Internet and reported an average information content score of 45.9% for this diagnosis. This is comparable to the mean content score of 49.2% in the current study. We believe this gives validation to the construction of the ACL reconstructionespecific content score. This score has not been used before but can now be used again in an interval fashion to assess the change of Internet-based information quality over time.
1099
ACL RECONSTRUCTION ON THE INTERNET Table 4. Summary of Website Scores
DISCERN JAMA Benchmark ACL Content Score
Fig 2. Distribution of websites by type.
Patients need to be made aware of the variability of health information on the Internet and the need for critical discerning reading.4 We should accept responsibility and take the time to educate our patients about how to identify high-quality sites and to direct them to appropriate sites.4,26 Finally, physicians could also become involved in posting content online themselves, although this admittedly takes time and certain skill sets. The website that ranked highest in this study (OrthoInfo by the American Academy of Orthopaedic Surgeons) shows us what can be done by translating
Fig 3. Quality and content scores by website type. (ACL, anterior cruciate ligament; JAMA, Journal of the American Medical Association.)
Type of Site
n Mean SD
Mean
SD
Mean
SD
Overall HONcode (present) HONcode (absent) Academic Commercial Discussion group Nonphysician Physician Unspecified
45 41.1 13.3 8 58.0 12.0
2.1 2.0
1.2 1.0
12.3 16.3
5.5 4.0
37 39.8 12.6
2.0
1.2
11.4
5.2
4 52.5 17.3 29 39.1 12.1 4 35.8 13.8
4.0 1.7 2.3
0 1.0 0.5
12.5 12.7 6.0
7.8 5.0 4.2
3.5 2.4 1.0
0.7 1.5 e
11.5 14.4 14
7.8 3.9 e
2 55 5 49 1 50
9.9 9.3 e
ACL, anterior cruciate ligament; HONcode, Health On the Net code; JAMA, Journal of the American Medical Association; SD, standard deviation.
evidence-based peer-reviewed studies into in-depth lay explanations and publishing them online.27 Limitations We acknowledge that this study has weaknesses. It is an assessment of quality at a single time point and does not represent any trenddindeed it would be invaluable to repeat the study after a period, as Nason et al.10 recently did on a different topic, to assess temporal trends. We have made assumptions based on best available information on the likely search engine used by a patient. Indeed, if an individual were to use a search engine other than those we have used, the results from this study would be less applicable. We used the search term ACL reconstruction rather than anterior cruciate ligament reconstruction because we believe most lay people would use the former. A search on the latter term matched on only 40% of unique websites, so results would differ. Finally, we have not determined the readability of any of the websites visited. Badarudeen
Fig 4. Association between outcome scores and Health On the Net (HONcode) certification. (ACL, anterior cruciate ligament; JAMA, Journal of the American Medical Association.)
1100
R. A. BRUCE-BRAND ET AL.
and Sabharwal28 have suggested that material on health care topics be stratified to reach the various reading levels of the patient audience.
Conclusions The quality of information available online regarding ACL reconstruction is of variable quality, with many websites returned by popular search engines omitting basic information regarding treatment options, risks, and prognosis. Commercial websites predominate, but academic sites provide higher quality content. The HONcode seal is a reliable indicator of website quality, and we can confidently recommend to our patients that they search for this marker.
References 1. Internet World Stats. Usage and population statistics. Available from: http://www.internetworldstats.com/. Accessed 4 Aug 2012. 2. Fox S, Rainie L, Horrigan J, et al. The online health care revolution: How the Web helps Americans take better care of themselves. Pew Internet & American Life Project. Available from: http://www.pewinternet.org/. Accessed August 4, 2012. 3. Stinson JN, Tucker L, Huber A, et al. Surfing for juvenile idiopathic arthritis: Perspectives on quality and content of information on the Internet. J Rheumatol 2009;36: 1755-1762. 4. Bichakjian CK, Schwartz JL, Wang TS, Hall JM, Johnson TM, Biermann JS. Melanoma information on the Internet: Often incompleteda public health opportunity? J Clin Oncol 2002;20:134-141. 5. Biermann JS, Golladay GJ, Greenfield ML, Baker LH. Evaluation of cancer information on the Internet. Cancer 1999;86:381-390. 6. Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Med 2010;8:59. 7. Kunst H, Groot D, Latthe PM, Latthe M, Khan KS. Accuracy of information on apparently credible websites: Survey of five common health topics. BMJ 2002;324:581-582. 8. Lee CT, Smith CA, Hall JM, Waters WB, Biermann JS. Bladder cancer facts: Accuracy of information on the Internet. J Urol 2003;170:1756-1760. 9. Meric F, Bernstam EV, Mirza NQ, et al. Breast cancer on the World Wide Web: cross sectional survey of quality of information and popularity of websites. BMJ 2002;324: 577-581. 10. Nason GJ, Baker JF, Byrne DP, Noel J, Moore D, Kiely PJ. Scoliosis-specific information on the internet: Has the “information highway” led to better information provision? Spine 2012;37:E1364-E1369. 11. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewordLet the reader and viewer beware. JAMA 1997;277:1244-1245.
12. Starman JS, Gettys FK, Capo JA, Fleischli JE, Norton HJ, Karunakar MA. Quality and content of Internet-based information for ten common orthopaedic sports medicine diagnoses. J Bone Joint Surg Am 2010;92:1612-1618. 13. Schindler OS. The story of anterior cruciate ligament reconstructiondPart 1. J Perioper Pract 2012;22:163-171. 14. Lyman S, Koulouvaris P, Sherman S, Do H, Mandl LA, Marx RG. Epidemiology of anterior cruciate ligament reconstruction: Trends, readmissions, and subsequent knee surgery. J Bone Joint Surg Am 2009;91:2321-2328. 15. Gottlob CA, Baker CL Jr, Pellissier JM, Colvin L. Cost effectiveness of anterior cruciate ligament reconstruction in young adults. Clin Orthop Relat Res 1999;367:272-282. 16. Cailliez J, Reina N, Molinier F, Chaminade B, Chiron P, Laffosse JM. Patient information ahead of anterior cruciate ligament reconstruction: Experience in a university hospital center. Orthop Traumatol Surg Res 2012;98: 491-498. 17. Koh HS, In Y, Kong CG, Won HY, Kim KH, Lee JH. Factors affecting patients’ graft choice in anterior cruciate ligament reconstruction. Clin Orthop Surg 2010;2:69-75. 18. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999;53:105-111. 19. Boyer C, Baujard V, Geissbuhler A. Evolution of health web certification through the HONcode experience. Stud Health Technol Inform 2011;169:53-57. 20. Netmarketshare.com. Search engine market share. Available from: www.netmarketshare.com. Accessed 4 Aug 2012. 21. Delince P, Ghafil D. Anterior cruciate ligament tears: Conservative or surgical treatment? A critical review of the literature. Knee Surg Sports Traumatol Arthrosc 2011;20: 48-61. 22. Tjoumakaris FP, Herz-Brown AL, Bowers AL, Sennett BJ, Bernstein J. Complications in brief: Anterior cruciate ligament reconstruction. Clin Orthop Relat Res 2011;470: 630-636. 23. Bernstein J, Ahn J, Veillette C. The future of orthopaedic information management. J Bone Joint Surg Am 2012;94: e95. 24. White RW, Horvitz E. Experiences with web search on medical concerns and self diagnosis. AMIA Annu Symp Proc 2009:696-700. 25. Tavare AN, Alsafi A, Hamady MS. Analysis of the quality of information obtained about uterine artery embolization from the internet. Cardiovasc Intervent Radiol 2012;35: 1355-1362. 26. Beredjiklian PK, Bozentka DJ, Steinberg DR, Bernstein J. Evaluating the source and content of orthopaedic information on the Internet. The case of carpal tunnel syndrome. J Bone Joint Surg Am 2000;82-A:1540-1543. 27. American Academy of Orthopaedic Surgeons. Orthoinfo. Your connection to expert orthopaedic information. Available from: http://orthoinfo.aaos.org/. Accessed August 10, 2012. 28. Badarudeen S, Sabharwal S. Assessing readability of patient education materials: Current role in orthopaedics. Clin Orthop Relat Res 2010;468:2572-2580.
Assessment of the Quality and Content of Information on Anterior Cruciate Ligament Reconstruction on the Internet Robert A. Bruce-Brand, M.R.C.S.I., Joseph F. Baker, M.R.C.S.I., Damien P. Byrne, Ph.D., Niall A. Hogan, F.R.C.S.I., and Tom McCarthy, F.R.C.S.I.
Abstract: The Internet has become a major source of health information for the public. However, there are concerns regarding the quality, accuracy, and currency of medical information available online. We assessed the quality of information about anterior cruciate ligament (ACL) reconstruction on the first 60 websites returned by the 4 most popular search engines. Each site was categorized by type and assessed for quality and validity using the DISCERN score, the Journal of the American Medical Association (JAMA) benchmark criteria, and a novel ACL reconstructionespecific content score. The presence of the Health On the Net Code (HONcode), a purported quality assurance marker, was noted. The quality of information on ACL reconstruction available online is variable, with many websites omitting basic information regarding treatment options, risks, and prognosis. Commercial websites predominate. Academic and allied health professional websites attained the highest DISCERN and JAMA benchmark scores, whereas physician sites achieved the highest content scores. Sites that bore the HONcode seal obtained higher DISCERN and ACL reconstruction content scores than those without this certification. The HONcode seal is a reliable indicator of website quality, and we can confidently advise our patients to search for this marker.
P
hysicians should be cognizant of the information their patients are likely to be accessing online and be able to steer them to high-quality websites. The Internet’s exponential growth has triggered an information revolution, and it has evolved into a major source of health information for the public. More than 273 million Americans, representing 78.6% of the population, now have Internet access.1 More than half use the World Wide Web to obtain health or medical information, and the majority of them seek health information online at least once a month.2 However, although the Internet represents an unprecedented method of sharing and accessing knowledge, there are growing concerns about the quality and currency of the medical information available. Several studies have
From the Department of Orthopaedic Surgery, St. James’s Hospital (R.A.B-B, J.F.B., N.A.H., T.M.); and Department of Orthopaedic Research, Orthopaedic Research & Innovation Foundation (D.P.B), Dublin, Ireland. The authors report that they have no conflicts of interest in the authorship and publication of this article. Received November 6, 2012; accepted February 1, 2013. Address correspondence to Robert A. Bruce-Brand, M.R.C.S.I., St James’s Hospital, James’s St, Dublin 8, Ireland. E-mail: [email protected] Ó 2013 by the Arthroscopy Association of North America 0749-8063/12726/$36.00 http://dx.doi.org/10.1016/j.arthro.2013.02.007
shown that many health sites contain misleading, inaccurate, and inappropriate information.3-12 Injury of the anterior cruciate ligament (ACL) is common in athletes, with an annual incidence of about 35 per 100,000 of the general population.13 It is one of the most commonly treated conditions of the knee.14 Estimated rates of ACL reconstruction exceed 175,000 per year in the United States.15 The Internet is an important source for patients seeking information about ACL reconstruction surgery. Cailliez et al.16 investigated patient knowledge ahead of ACL reconstruction and found that 40% of patients had researched on the Internet for surgery-related information. Koh et al.17 examined factors affecting patients’ graft choice in ACL reconstruction and found that the Internet was the most important contributor in 21.7% of cases. However, we are not aware of any study that has determined the quality of information about ACL reconstruction found on the Internet. The aim of this study, therefore, was to assess the quality of information about ACL reconstruction on the Internet using recognized scoring systems, identification of quality markers, and a novel specific score. Scoring systems included the Journal of the American Medical Association (JAMA) benchmark and DISCERN criteria.11,18 Initiatives such as the Health On the Net code (HONcode) identify websites that agree to comply with a code
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol 29, No 6 (June), 2013: pp 1095-1100
1095
1096
R. A. BRUCE-BRAND ET AL.
of ethics to provide quality objective and transparent medical information.19
Searching on the Web We used the search term ACL reconstruction on the 4 most popular search engines: Google, Yahoo!, Bing, and Ask. Google is the dominant search engine, composing 83.06% of the total market share, followed by Yahoo! (6.86%), Bing (4.27%), and Ask (0.51%) (figures as of August, 2012).20 We had planned to review the first 60 websites encountered during an Internet search. To reflect the disproportionate use of the various available search engines, we analyzed the first 30 of approximately 985,000 websites returned by Google (www .google.com) and 10 each from Yahoo! (www.search .yahoo.com), Bing (www.bing.com), and Ask (www .ask.com). The total number of hits returned by the search is shown in Table 1. Searches were performed on August 10, 2012. Duplicate websites and those that were inaccessible were excluded from analysis (Fig 1). The first part of the analysis involved determining the type of website. We categorized sites as academic, physician, nonphysician (allied health professionals such as physical therapists, occupational therapists, and alternative medical providers), commercial, media, discussion groups/social networks, and unspecified. These classifications were based on those described in previous work, with the addition of the discussion groups/social media class to reflect the growing trend in sites such as YouTube (www.youtube.com), Facebook (www.facebook.com), and Twitter (www.twitter.com).10,12 Academic websites included those affiliated with a university or medical society. Websites were considered commercial if they displayed advertisements or included products for sale. Physician websites included professional sites for individual physicians as well as practitioner groups not affiliated with an academic institution. Media websites were nonmedical news-oriented sites. Any website that did not fit the preceding categories was classed as unspecified. Websites were then assessed for quality and validity using the DISCERN score, the JAMA benchmark criteria, and an ACL reconstructionespecific content score. They were also assessed for the presence or absence of the HONcode certification.11,18,19 DISCERN, developed by an expert group in the United Kingdom, is an instrument designed to judge the quality of written consumer health information about treatment choices. It determines the quality of a publication Table 1. Search Results for ACL Reconstruction Search Engine Google Yahoo! Bing Ask
Approximate Number of Search Results 985,000 6,450,000 1,710,000 Not available
Fig 1. Internet search flow diagram.
based on 16 questions. The first 8 questions concern the reliability of the publication, and the subsequent 7 questions address specific details of the information about treatment choices. The last question is an overall quality rating of the website. Silberg et al.11 described 4 core criteria to allow readers to determine whether what they were reading was credible, reasonable, or useful. Also known as the JAMA benchmark criteria, they assess authorship, attribution, disclosure, and currency. The authorship criterion requires that the website provide details of authors and contributors, their affiliations, and relevant credentials. Attribution entails listing references and sources for all content and indicating any relevant copyright information. Currency refers to the provision of the dates that content was posted and updated, and disclosure requires that website ownership be prominently and fully disclosed, along with any sponsorship, advertising policies, and potential conflicts of interest. To assess the informational value of each website, we developed an ACL reconstructionespecific content score comprising 25 factors (Table 2). This was done in consultation with sports surgeryetrained consultants and by referencing current peer-reviewed literature.21,22 One point was allocated for the mention of each of the predefined terms that related to general characteristics of the condition, its prognosis, treatment options, and complications. Sites were then scored from 0 to 25, with 25 indicating a site of maximal quality. The Health On the Net Foundation is a nonprofit nongovernmental organization that was established in 1996 and established ethical standards for publishing
1097
ACL RECONSTRUCTION ON THE INTERNET Table 2. ACL ReconstructioneSpecific Content Score General Aspects
Management
Complications
Age Sex Mechanisms Functional disability
Observation Infection Physical therapy Graft failure Bracing Knee pain Reconstruction: Degenerative knee hamstring Functional Reconstruction: requirements patellar tendon Meniscal injury Reconstruction: allograft Timing of surgery Single bundle Anesthesia Double bundle Rehabilitation Anatomic reconstruction Return to sports/work Meniscal repair Revision reconstruction
NOTE. One mark was allocated for each point mentioned on the website.
medical and health-related information on the Internet. The HONcode seal accredits websites that agree to comply with listed standards and publish transparent healthrelated information. For each website, we checked for the presence and validity of the HONcode seal. Statistical analysis was performed using SPSS Statistics, version 17 (SPSS, Chicago, IL). The level of significance was set at 5%. A one-way analysis of variance was used to determine whether website type predicted higher DISCERN, JAMA benchmark, and ACL reconstruction content scores. Tukey multiple comparison methods were used to compare the different means. To determine whether HONcode certification predicted higher scores, 2 sample t tests were used when data was normally distributed, whereas Mann-Whitney U tests were used when data was not normally distributed.
Results A total of 45 unique websites was analyzed (Table 3). Fig 2 shows the breakdown according to website type. The majority were commercial (n ¼ 29 [64%]). Eight (17.8%) of the websites were HONcode certified. The mean overall scores by website type are illustrated in Fig 3. DISCERN Scores The mean DISCERN score for all websites was 41.1 13.3 of a maximum of score of 80 (range, 15 to 78). The highest-scoring website was an academic site. Nineteen websites (42.2%) were scored as being of poor quality, with serious or extensive shortcomings. Only 4 sites attained scores greater than 60, representing excellent quality with minimal shortcomings. The academic and nonphysician website types had the highest overall DISCERN scores (Table 4). However, differences in DISCERN scores were not significantly different between the website types (P > .05). Websites that were HONcode certified had significantly higher mean DISCERN scores (58.0 v 39.8; P ¼ .016).
JAMA Benchmark Criteria The mean JAMA benchmark criteria score was 2.1 1.2. Ten sites had maximal scores. There was a significant difference between the groups (P ¼ .001), with the academic and nonphysician (allied health professionals) sites scoring highest. ACL ReconstructioneSpecific Content Score The highest content score was obtained by an academic site with a score of 24 of 25. One website, of the discussion group type, received 0 points. The mean overall score was 12.29 5.49. The physician-type websites had the highest overall content score, with a mean of 14.4. However, intergroup differences were not significant. HONcode Certification The association between outcome scores and HONcode certification is shown in Fig 4. Websites that bore the HONcode seal obtained higher DISCERN and ACL reconstruction content scores than those without this certification (Table 4). However, JAMA benchmark criteria were not significantly different (2.0 v 2.0; P ¼ .14).
Discussion The Internet has revolutionized the dissemination of information, and patients can access boundless information on any subject, including health.2,12,23 However, unlike traditional media, it allows virtually unregulated publication of information that is available on the widest possible scale. The challenge to the Internet “surfer” therefore is to find credible reliable information among the unregulated mountains of hypertext. The aim of this study was to determine the quality of Internet-based information on ACL reconstruction using recognized quality control measures. Our findings indicate that the websites likely to be accessed by patients are of variable quality. The mean DISCERN score we recorded represents information of only moderate quality, with potentially important but not serious shortcomings. This has the potential to detrimentally alter the patient-physician consultation as well as cause undue anxiety for the patient. The term cyberchondria has been proposed to represent the possible mental state after reading inaccurate information based on the Internet.24 Consistent with previous studies, we found that academic institutionecreated sites provided better information quality.10 However, we also noted that nonphysician (allied health professional) sites achieved the highest mean DISCERN score; this is an inconsistency with previous comparable studies but may result from the small number of nonphysician sites encountered in this study.10 Patients searching the Internet can reliably be directed to look for sites provided by academic institutions.
1098
R. A. BRUCE-BRAND ET AL.
Table 3. List of Unique Websites Evaluated Search Engine Google
Yahoo!
Bing
Ask
Website 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 45.
http://en.wikipedia.org/wiki/Anterior_cruciate_ligament_reconstruction http://www.youtube.com/watch?v¼q96M0jRqn7k http://www.nlm.nih.gov/medlineplus/ency/article/007208.htm http://www.hipandkneesurgery.ie/ligament-reconstruction.html http://orthoinfo.aaos.org/topic.cfm?topic¼A00297 http://www.orthogate.org/patient-education/knee/anterior-cruciate-ligament-injuries.html http://orthopedics.about.com/od/aclinjury/tp/acl.htm http://www.webmd.com/a-to-z-guides/anterior-cruciate-ligament-acl-surgery http://www.macdarabutler.com/acl-reconstruction-patella-vs-hamstring-graft/ http://www.aclreconstruction.org/ http://www.mayoclinic.com/health/acl-reconstruction/MY00941/ http://www.scoi.com/aclrecon.htm http://orthopedics.about.com/cs/aclrepain/a/acldecision.htm http://www.orthosports.com.au/content_common/pg-acl-reconstruction.seo http://www.sportsmedalabama.com/patient-education/knee-mcl-acl-reconstruction.html http://drmillett.com/acl-repair-anterior-cruciate-ligament http://www.biomet.com/patients/sportsMedicine/aclReconstruction.cfm http://www.sportmed.ucalgary.ca/ACL http://www.hopkinsmedicine.org/healthlibrary/test_procedures/orthopaedic/knee_ligament_repair_92,P07675/ http://myaclreconstruction.blogspot.com/ http://www.hipsknees.info/flash5/acl_patellor_final.pdf http://www.sharonregional.com/acl-reconstruction.htm http://www.hopkinsortho.org/acl.html http://www.fixknee.com/ACLRecon/ACL%20Recontruction2011.html http://www.aaos.org/news/aaosnow/apr12/cover1.asp http://www.aucklandorthopaedicpractice.co.nz/ligament-reconstruction.html http://www.pitt.edu/wmechbio/publications/graft.pdf http://www.iskinstitute.com/pdf/pdf_forms/ACL%20Reconstruction%20Book.pdf http://www.orlive.com/broward-health/videos/anterior-cruciate-ligament-acl-reconstruction-surgery http://isgweb.aorn.org/ISGWeb/downloads/CEA11002-4055.pdf http://www.bmihealthcare.co.uk/treatment/treatmentsdetail?p_name¼ACL%20reconstruction%20 surgery&p_treatment_id¼307 http://ehealthmd.com/acl-tears/recovery-after-acl-reconstruction http://www.arthroscopy.com/sp05025.htm http://footballsfuture.com/phpBB2/viewtopic.php?p¼14564779 http://www.pamf.org/sports/king/anterior_cruciate.html http://www.kneeclinic.info/problems_acl.php http://www.uwmsk.org/residentprojects/aclreconstruction.html http://www.knee1.com/Education_Center/Therapies/ACL_Reconstruction http://ask.healthline.com/health/acl-reconstruction http://www.ehow.com/facts_5610921_acl-reconstruction.html http://www.anteriorcruciateligament.co.uk/ http://www.londonkneeclinic.com/knee-problems/ligament-injuries#ACL http://www.noc.nhs.uk/hipandknee/information/documents/acl-reconstruction.pdf http://www.oxfordradcliffe.nhs.uk/forpatients/090427patientinfoleaflets/100419acl.pdf http://www.bupa.co.uk/individuals/health-information/directory/a/acl
Our study indicates that for Internet searches on ACL reconstruction, the HONcode certification reliably identifies websites with significantly higher quality and content scores. This mirrors the findings of similar studies investigating the quality of other health care topics.6,12,25 This is reassuring, and as physicians we can direct our patients to look for this quality assurance marker when they search the Internet. Starman et al.12 found that the quality and content of Internet-based information for 10 common orthopaedic sports medicine diagnoses were highly variable, consistent with our findings, with commercial sites dominating search results.12 With the majority of search engine results in this study being commercial sites, there is clear
potential for bias in the information presented. It is necessary for physicians to be aware that their patients may be misinformed and to spend time correcting patients’ perceptions. Starman et al.12 included ACL tear as one of the 10 common orthopaedic sports medicine diagnoses they analyzed on the Internet and reported an average information content score of 45.9% for this diagnosis. This is comparable to the mean content score of 49.2% in the current study. We believe this gives validation to the construction of the ACL reconstructionespecific content score. This score has not been used before but can now be used again in an interval fashion to assess the change of Internet-based information quality over time.
1099
ACL RECONSTRUCTION ON THE INTERNET Table 4. Summary of Website Scores
DISCERN JAMA Benchmark ACL Content Score
Fig 2. Distribution of websites by type.
Patients need to be made aware of the variability of health information on the Internet and the need for critical discerning reading.4 We should accept responsibility and take the time to educate our patients about how to identify high-quality sites and to direct them to appropriate sites.4,26 Finally, physicians could also become involved in posting content online themselves, although this admittedly takes time and certain skill sets. The website that ranked highest in this study (OrthoInfo by the American Academy of Orthopaedic Surgeons) shows us what can be done by translating
Fig 3. Quality and content scores by website type. (ACL, anterior cruciate ligament; JAMA, Journal of the American Medical Association.)
Type of Site
n Mean SD
Mean
SD
Mean
SD
Overall HONcode (present) HONcode (absent) Academic Commercial Discussion group Nonphysician Physician Unspecified
45 41.1 13.3 8 58.0 12.0
2.1 2.0
1.2 1.0
12.3 16.3
5.5 4.0
37 39.8 12.6
2.0
1.2
11.4
5.2
4 52.5 17.3 29 39.1 12.1 4 35.8 13.8
4.0 1.7 2.3
0 1.0 0.5
12.5 12.7 6.0
7.8 5.0 4.2
3.5 2.4 1.0
0.7 1.5 e
11.5 14.4 14
7.8 3.9 e
2 55 5 49 1 50
9.9 9.3 e
ACL, anterior cruciate ligament; HONcode, Health On the Net code; JAMA, Journal of the American Medical Association; SD, standard deviation.
evidence-based peer-reviewed studies into in-depth lay explanations and publishing them online.27 Limitations We acknowledge that this study has weaknesses. It is an assessment of quality at a single time point and does not represent any trenddindeed it would be invaluable to repeat the study after a period, as Nason et al.10 recently did on a different topic, to assess temporal trends. We have made assumptions based on best available information on the likely search engine used by a patient. Indeed, if an individual were to use a search engine other than those we have used, the results from this study would be less applicable. We used the search term ACL reconstruction rather than anterior cruciate ligament reconstruction because we believe most lay people would use the former. A search on the latter term matched on only 40% of unique websites, so results would differ. Finally, we have not determined the readability of any of the websites visited. Badarudeen
Fig 4. Association between outcome scores and Health On the Net (HONcode) certification. (ACL, anterior cruciate ligament; JAMA, Journal of the American Medical Association.)
1100
R. A. BRUCE-BRAND ET AL.
and Sabharwal28 have suggested that material on health care topics be stratified to reach the various reading levels of the patient audience.
Conclusions The quality of information available online regarding ACL reconstruction is of variable quality, with many websites returned by popular search engines omitting basic information regarding treatment options, risks, and prognosis. Commercial websites predominate, but academic sites provide higher quality content. The HONcode seal is a reliable indicator of website quality, and we can confidently recommend to our patients that they search for this marker.
References 1. Internet World Stats. Usage and population statistics. Available from: http://www.internetworldstats.com/. Accessed 4 Aug 2012. 2. Fox S, Rainie L, Horrigan J, et al. The online health care revolution: How the Web helps Americans take better care of themselves. Pew Internet & American Life Project. Available from: http://www.pewinternet.org/. Accessed August 4, 2012. 3. Stinson JN, Tucker L, Huber A, et al. Surfing for juvenile idiopathic arthritis: Perspectives on quality and content of information on the Internet. J Rheumatol 2009;36: 1755-1762. 4. Bichakjian CK, Schwartz JL, Wang TS, Hall JM, Johnson TM, Biermann JS. Melanoma information on the Internet: Often incompleteda public health opportunity? J Clin Oncol 2002;20:134-141. 5. Biermann JS, Golladay GJ, Greenfield ML, Baker LH. Evaluation of cancer information on the Internet. Cancer 1999;86:381-390. 6. Kaicker J, Debono VB, Dang W, Buckley N, Thabane L. Assessment of the quality and variability of health information on chronic pain websites using the DISCERN instrument. BMC Med 2010;8:59. 7. Kunst H, Groot D, Latthe PM, Latthe M, Khan KS. Accuracy of information on apparently credible websites: Survey of five common health topics. BMJ 2002;324:581-582. 8. Lee CT, Smith CA, Hall JM, Waters WB, Biermann JS. Bladder cancer facts: Accuracy of information on the Internet. J Urol 2003;170:1756-1760. 9. Meric F, Bernstam EV, Mirza NQ, et al. Breast cancer on the World Wide Web: cross sectional survey of quality of information and popularity of websites. BMJ 2002;324: 577-581. 10. Nason GJ, Baker JF, Byrne DP, Noel J, Moore D, Kiely PJ. Scoliosis-specific information on the internet: Has the “information highway” led to better information provision? Spine 2012;37:E1364-E1369. 11. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewordLet the reader and viewer beware. JAMA 1997;277:1244-1245.
12. Starman JS, Gettys FK, Capo JA, Fleischli JE, Norton HJ, Karunakar MA. Quality and content of Internet-based information for ten common orthopaedic sports medicine diagnoses. J Bone Joint Surg Am 2010;92:1612-1618. 13. Schindler OS. The story of anterior cruciate ligament reconstructiondPart 1. J Perioper Pract 2012;22:163-171. 14. Lyman S, Koulouvaris P, Sherman S, Do H, Mandl LA, Marx RG. Epidemiology of anterior cruciate ligament reconstruction: Trends, readmissions, and subsequent knee surgery. J Bone Joint Surg Am 2009;91:2321-2328. 15. Gottlob CA, Baker CL Jr, Pellissier JM, Colvin L. Cost effectiveness of anterior cruciate ligament reconstruction in young adults. Clin Orthop Relat Res 1999;367:272-282. 16. Cailliez J, Reina N, Molinier F, Chaminade B, Chiron P, Laffosse JM. Patient information ahead of anterior cruciate ligament reconstruction: Experience in a university hospital center. Orthop Traumatol Surg Res 2012;98: 491-498. 17. Koh HS, In Y, Kong CG, Won HY, Kim KH, Lee JH. Factors affecting patients’ graft choice in anterior cruciate ligament reconstruction. Clin Orthop Surg 2010;2:69-75. 18. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999;53:105-111. 19. Boyer C, Baujard V, Geissbuhler A. Evolution of health web certification through the HONcode experience. Stud Health Technol Inform 2011;169:53-57. 20. Netmarketshare.com. Search engine market share. Available from: www.netmarketshare.com. Accessed 4 Aug 2012. 21. Delince P, Ghafil D. Anterior cruciate ligament tears: Conservative or surgical treatment? A critical review of the literature. Knee Surg Sports Traumatol Arthrosc 2011;20: 48-61. 22. Tjoumakaris FP, Herz-Brown AL, Bowers AL, Sennett BJ, Bernstein J. Complications in brief: Anterior cruciate ligament reconstruction. Clin Orthop Relat Res 2011;470: 630-636. 23. Bernstein J, Ahn J, Veillette C. The future of orthopaedic information management. J Bone Joint Surg Am 2012;94: e95. 24. White RW, Horvitz E. Experiences with web search on medical concerns and self diagnosis. AMIA Annu Symp Proc 2009:696-700. 25. Tavare AN, Alsafi A, Hamady MS. Analysis of the quality of information obtained about uterine artery embolization from the internet. Cardiovasc Intervent Radiol 2012;35: 1355-1362. 26. Beredjiklian PK, Bozentka DJ, Steinberg DR, Bernstein J. Evaluating the source and content of orthopaedic information on the Internet. The case of carpal tunnel syndrome. J Bone Joint Surg Am 2000;82-A:1540-1543. 27. American Academy of Orthopaedic Surgeons. Orthoinfo. Your connection to expert orthopaedic information. Available from: http://orthoinfo.aaos.org/. Accessed August 10, 2012. 28. Badarudeen S, Sabharwal S. Assessing readability of patient education materials: Current role in orthopaedics. Clin Orthop Relat Res 2010;468:2572-2580.