- Email: [email protected]
Myelomeningocele Information on the Internet is Accessible and of Variable Quality, and Requires a High Reading Level
Myelomeningocele Information on the Internet is Accessible and of Variable Quality, and Requires a High Reading Level Jonathan Bergman, Ramdev Konijeti and Steven E. Lerman From the Department of Urology, Division of Pediatric Urology, David Geffen School of Medicine at University of California Los Angeles, Los Angeles, California
Purpose: We assessed the accessibility, readability and quality of myelomeningocele information on the Internet. Materials and Methods: We entered the term “spina bifida” into the Google, Yahoo! and Microsoft Network search engines, and stored the first 100 links from each Web site. A total of 164 unique Web sites remained for analysis, of which 159 were classified as relevant. Relevant Web sites were considered to have relevant content if more than 50% of the text was directly relevant to the disease, and to have relevant educational content if more than 50% of the text was aimed at relaying educational information. Readability was assessed using the Flesch-Kincaid Grade Level scale. The quality of the 159 Web sites with relevant content was assessed using American Public Health Association Criteria for Assessing Health Information on the Internet. Six criteria were assessed, namely credibility, content, links, design, interactivity and caveats. Web sites were analyzed by 2 separate medical doctors, with each reviewer blinded to the findings of the other. A weighted kappa statistic was used to calculate interrater reliability. Results: Of 159 relevant sites 146 (91.8%) had relevant content and 122 (76.7%) had relevant educational content. The average Flesch-Kincaid reading level was 10.9 (range 6 to 12). Quality was assessed on a 3-point scale, with 1 denoting poor quality, 2 fair and 3 good. Average scores were 1.92 for credibility, 1.88 for content, 2.29 for links, 2.53 for design, 1.99 for interactivity and 2.90 for caveats. The weighted kappa statistic for interrater reliability was 0.83. Conclusions: Myelomeningocele information on the Internet is relatively accessible, requires a high reading level for comprehension and is of variable quality. Key Words: meningomyelocele, spina bifida occulta, Internet, quality control
yelomeningocele currently affects approximately 70,000 people living in the United States.1 Before the United States Public Health Service folic acid mandate myelomeningocele affected approximately 1 in 1,600 pregnancies and 1 in 2,000 live births. Even after the mandate this condition is estimated to affect 1 in 2,440 pregnancies and 1 in 3,000 live births.1,2 Children with myelomeningocele have incomplete spinal cord development, with subsequent neurological deficits below the lesion. Associated impairments include brain malformations, hydrocephalus, neuropsychological difficulties, difficulties in bowel and bladder function, and difficulty in ambulation.3 While functional status with myelomeningocele varies among patients, coordination of care is complex and comprehension of the disease intricacies is a challenge for most families.4 It is estimated that up to 5% of all World Wide Web searches are health related, with up to 7 million health related searches conducted each day.5 Several reports have questioned the accuracy and readability of health related Web sites, and tools have been developed to study the quality, readability and suitability of health related sites.6,7 Internet information regarding myelomeningocele has yet to be studied. We assessed the accessibility, readability and quality of myelomeningocele information on the Internet.
M
Submitted for publication August 16, 2006.
0022-5347/07/1773-1138/0 THE JOURNAL OF UROLOGY® Copyright © 2007 by AMERICAN UROLOGICAL ASSOCIATION
MATERIALS AND METHODS Search engine selection was based on the Nielsen NetRatings, which showed that Google (46.2% of total online searches), Yahoo! (22.5%) and Microsoft Network (12.6%) were by far the most commonly used search engines.8 Between January 11 and January 13, 2006 trained searchers entered the term “spina bifida” into each of the 3 search engines, and the first 100 links from each Web site were stored. The total number of hits exceeded 200,000 on Google, 2,000,000 on Yahoo and 500,000 on Microsoft Network. Therefore, the first 100 links represented approximately 0.011% of the total number of hits but included the first sites to which patients would be linked when searching. This method of assessing multiple Web sites was consistent with previous studies.9 Web sites from academic, general health, personal, governmental, private profit, nonprofit and institutional sites were included, whereas advertisements linked to the results page were excluded. The figure outlines the process followed to categorize Web sites. Duplicate Web sites were excluded, leaving 164 unique Web sites for analysis. The links were then followed and classified as relevant if the search term was present in the text of the first electronic page to which the link led, resulting in 159 relevant Web sites. The 5 irrelevant Web sites were personal advertisement pages where the term “spina bifida” could be found in the text of 1 of the advertisements. Relevant Web sites were coded for content, and were considered to have relevant content if more than 50% of the
1138
Vol. 177, 1138-1142, March 2007 Printed in U.S.A. DOI:10.1016/j.juro.2006.10.019
MYELOMENINGOCELE INFORMATION ON INTERNET
1139
Web site classification process. MSN, Microsoft Network
text was directly relevant to the disease. In addition, Web sites with relevant content were classified as having relevant educational content if more than 50% of the text was aimed at relaying educational information to patients or families. The number of mouse clicks needed to reach the main educational page was recorded. The method of identifying accessible Web sites was consistent with previous studies.9,10 Readability was assessed using the Flesch-Kincaid Grade Level scale by copying all relevant text from the Web page and subsequent links into Microsoft Word, performing a spelling and grammar check, and recording the Flesch-Kincaid reading level.11 The quality of the 159 Web sites with relevant content was assessed using American Public Health Association Criteria for Assessing Health Information on the Internet.12 Six criteria were assessed, namely credibility, including the source, currency, relevance/utility, editorial review process for the information and financial disclosure; content, including accuracy, completeness and inclusion of appropriate disclaimers; links, including the relevance and comprehensiveness of links provided; design, including accessibility, navigability and internal search capacity; interactivity, including feedback mechanisms and means for exchange of information among users; and caveats, including clarification of whether the primary function of the site was to provide content or market products and services. Web sites were analyzed by 2 separate medical doctors (JB and RK), with each reviewer blinded to the findings of the other. A weighted kappa statistic was used to calculate interrater reliability. RESULTS Of 159 relevant sites 146 (91.8%) had relevant content and 122 (76.7%) had relevant educational content. Appendix 1 lists the 159 relevant Web sites. The average number of links needed to arrive at the main educational Web page was
0.42 (range 0 to 3), and the average Flesch-Kincaid reading level was 10.9 (6 to 12). Quality, including credibility, content, links, design, interactivity and caveats, was assessed on a 3-point scale, with 1 denoting poor quality, 2 fair and 3 good. The average scores were 1.92 for credibility, 1.88 for content, 2.29 for links, 2.53 for design, 1.99 for interactivity and 2.90 for caveats. The weighted kappa statistic for interrater reliability was 0.83. DISCUSSION Our findings addressed the accessibility, readability and quality of myelomeningocele information on the Internet. Accessibility findings were encouraging, with 159 of 164 Web sites (97%) having relevant information, of which 146 (91.8%) had relevant content and 122 (76.7%) had relevant educational content. These results are significantly better than those of previous accessibility studies performed for other medical problems.13–15 The number of links needed to arrive at educational information (0.42) suggests that myelomeningocele Web sites can be navigated with relative ease. Of the 6 quality indicators studied caveats rated most favorably, suggesting Web sites were clear in highlighting their primary function. Site design also rated well, suggesting that the accessibility, navigability and internal search capacity were fairly good. This finding reflected significant improvement over recent similar analyses for other diseases.4,6,7,9,13,15 Although the availability and relevance of links were above average, they surprisingly were inferior to design and caveats. Credibility, content and interactivity all rated below “fair.” Shortcomings in credibility were due to deficiencies in currency, relevance, utility and editorial review process. Black and Penson recently studied prostate cancer on the Internet and found similar deficiencies in credibil-
1140
MYELOMENINGOCELE INFORMATION ON INTERNET
ity, especially in currency of information.16 Deficiencies in content suggest the information patients and their families read on the Internet may be inaccurate or incomplete, or contain inappropriate disclaimers. Similar problems have been highlighted by others for manifold medical problems, although improvement with time has yet to be demonstrated.13–15 The failure of the Web sites to provide good interactivity is especially relevant for a disease such as myelomeningocele, where the lay public may be less informed than they would be regarding higher profile diseases such as prostate cancer or breast cancer. Children with myelomeningocele are at increased risk for poor psychosocial development, and interacting with peers at school and in other settings is vital to their psychosocial adjustment.17 Readability addresses the ability to comprehend the meaning of written text. The average adult in the United States reads at a 5th to 8th grade level, with wide variability.18 The Flesch-Kincaid Grade Level scale evaluates readability based on average sentence length and average number of syllables per word, assigning 3rd to 12th grade reading levels. This scale has been validated and used to assess readability for more than 40 years.11 Consistent with other studies, we found that the reading level of the Web sites reviewed (10.9) was significantly higher than the reading skill of most potential readers. Clearly, not all Web sites are created equal, and some are highly accessible and readable, and of superior quality. In 2001 Pautler et al19 recommended that national and international societies, including the American Urological Association, create a list of approved or endorsed Web sites, a recommendation that appears to be cogent today, as patients are increasingly turning to the Web for health information but with questionably improved efficiency. The American Medical Association has published recommendations for Web site design, although a mechanism needs to be developed to assess the degree to which sites abide by the recommendations.20 Ideally, Web sites would be easily accessible, with relevant educational content that is easily navigable with a minimal number of mouse clicks. They would have an internal search capacity as well as a mechanism for user feedback and interactivity among readers. Information would be current and appropriately referenced. Importantly, text would be written at no higher than an 8th grade reading level, consistent with the ability of the average reader. Links would be provided to academic, nonprofit, self-help and group help Web sites. Appendix 2 lists 10 of the highly ranked myelomeningocele Web sites. Our study has several limitations. The quality of information was subjectively assessed on a 3-point scale, although the weighted kappa statistic (0.83) would suggest the sites were reliably graded, with minimal interobserver variability. Web sites studied were chosen by their ranking on 3 separate search engines, not necessarily based on frequency of patient or family use (although this is 1 of the factors programmed into each of the 3 search engines). However, our analysis of 300 sites, of which 164 were unique, is more comprehensive than most studies performed for other diseases. Our study was further limited by the fact that the total number of Web sites analyzed represented only 0.011% of the total hits. However, the analysis included the top 100 hits from each of the 3 search engines, and, therefore, was
most relevant to the sites patients or families were likely to explore when searching for information. Future studies looking at physician knowledge of the Web sites patients are visiting would be interesting. Developing a ranking system, or index, would help direct patients toward useful and accurate Web sites. Studying whether the reading level of Web sites can be decreased without sacrificing the quality or accessibility of the information would also be useful. In conclusion, myelomeningocele information on the Internet is accessible, requires a high reading level for sufficient comprehension, is of variable quality and can be navigated with relative ease. APPENDIX 1 159 Relevant Web Sites http://www.geocities.com/HotSprings/2474/ http://www.iashah.ie/ http://nyneurosurgery.org/spinabifida_intro.htm http://www.illinoisspinabifidaassociation.com/ http://www.sbhac.ca/index.php?page⫽main http://www.asbha.org.au/ http://cpmenet.columbia.edu/dept/nsg/PNS/SpinaBifida.html http://www.sbaalbany.org/ http://www.dpo.uab.edu/⬃birmie/sb.htm http://rarediseases.about.com/cs/spinabifida/a/101902.htm http://www.emedicinehealth.com/articles/34669-1.asp http://www.ifglobal.org/home.asp?lang⫽1&main⫽1 http://en.wikipedia.org/wiki/Spina_bifida http://montekids.org/programs/spinabifida/ http://www.tvcc.on.ca/gateway.php?id⫽31&cid⫽2&PHPSESSID⫽ 594a36ad38aebb1580d212e4a492cf47 http://www.childrensmemorial.org/depts/motionanalysis/conditions/ spinabifida.asp http://www.webmd.com/hw/raising_a_family/hw169958.asp? lastselectedguid⫽{5FE84E90-BC77-4056-A91C-9531713CA348} http://www.ninds.nih.gov/disorders/spina_bifida/detail_spina_bifida.htm http://www.gillettechildrens.org/default.efm/PID⫽1.3.18 http://www2.vhihealthe.com/article/gale/100084077 http://www.cnn.com/HEALTH/library/DS/00417.html http://www.mydr.com.au/default.asp?article⫽3571 http://www.marchofdimes.com/pnhec/4439.asp http://www.mayoclinic.com/health/spina-bifida/DS00417 http://208.62.55.81/aboutus.ctm http://www.childrenshospital.org/az/Site1062/mainpageS1062P0.html http://www.chkd.org/Neurology/spibif.asp http://www.fetalcarecenter.org/medicine/fetal-surgery/myelomeningocelesurgery.htm http://kidshealth.org/parent/system/ill/spina_bifida.html http://www.asbah.org/ http://checksutterfirst.org/spine/spina.html http://health.yahoo.com/ency/healthwise/hw169956 http://www.emedicine.com/orthoped/topic557.htm http://www.findarticles.com/p/articles/mi_g2601/is_0012/ai_2601001279 http://www.healthatoz.com/healthatoz/Atoz/ency/spina_bifida.jsp http://groups.msn.com/PainIssuesForPersonsWithSpinaBifida http://www.fetal-surgery.com/ http://www.geocities.com/enchantedforest/cottage/1109/ http://www.spinabifida.asn.au/ http://www.sbatx.org/ http://www.sbhabc.org/ http://www.sbac.org/ http://spinabifida.mjsd.ca/info.htm http://www.chesapeakespinabifida.org/facts.htm http://www.juliasjourney.com/ http://www.spinabifidainfo.nl/ www.fortunecity.com/millenium/plumpton/268/sb.htm http://www.azspinabifida.org/ http://www.sbawa.asn.au/ http://sbahgc.org/faq/index.cfm http://www.spinabifidaofgeorgia.org/ http://www.sbahgc.org/ http://webess25.micromedex.com/content/DiseaseDex/001258.htm http://www.cynsspot.com/SpinaBifida.htm http://www.spinabifidamoms.com/english/about.html http://www.sbhao.on.ca/ http://ibis-birthdefects.org/start/ntdfact.htm (appendix continued)
MYELOMENINGOCELE INFORMATION ON INTERNET APPENDIX 1 continued
APPENDIX 1 continued
159 Relevant Web Sites
159 Relevant Web Sites
http://www.keepkidshealthy.com/welcome/conditions/spinabifida.html http://www.medicinenet.com/spina_bifida_and_anencephaly/article.htm http://faculty.washington.edu/chudler/spibi.html http://www.specialchild.com/archives/dz-005.html http://www.ssba.org.uk/ http://www.spineuniverse.com/displayarticle.php/article234.html www.drgreene.com/21_1191.html http://www.msbaweb.org/ http://www.uromed.com/?kwov⫽children_with_spina_bifida&OVRAW⫽ spina%20bifida&OVKEY⫽child%20with%20spina%20bifida&OVMTC⫽ advanced http://www.nichcy.org/pubs/factshe/fs12txt.htm http://www.bbe.co.uk/health/conditions/spinabifida2.shtml http://www.intelihealth.com/IH/ihtIH/WSIHW000/9339/9619.html http://stephblog.com/?cat⫽17 http://members.tripod.com/⬃imaware/sb.html http://www.hopkinsmedicine.org/orthopedicsurgery/spinabifida.html http://ymghealthinfo.org/content.asp?page⫽P02624 http://kidshealth.org/kid/health_problems/brain/spina_bifida.html http://www.irishhealth.com/?level⫽4&con⫽471 http://www.drgreene.com/21_1191.html http://www.neurosurgerytoday.org/what/patient_e/spina.asp http://ww2.kgw.com/Global/story.asp?S⫽1230739&nav⫽menu296_5 http://www.netwellness.org/ency/article/001558.htm http://www.kidshealth.org/kid/health_problems/bone/spina_bifida.html http://www.nlm.nih.gov/medlineplus/ency/article/001558.htm http://www.hmc.psu.edu/healthinfo/s/spinabifida.htm http://www.ivillage.co.uk/pregnancyandbaby/baby/babyhealth/articles/ 0,30_163464,00.html http://www.geocities.com/SBAofgpa/ http://www.geocities.com/Heartland/Acres/1297/spinabifida.html http://www.eplibrary.com/spinabifida/ http://www.exceptionalteaching.com/bifida.html http://www3.ns.sympatico.ca/spina.bifida/ http://www.srhsb.org/ http://latexallergylinks.tripod.com/sb.html http://users.iconz.co.nz/apeacock/mobility.html http://www.sbak.org/ http://www.sbfv.org.au/ http://www.sbatsr.org/ http://www.sbawp.org/ http://www.spinabifidalaw.com/ http://www.sbagno.org/ http://www.waisman.wise.edu/⬃rowley/sb_kids.htmlx http://www.kumc.edu/gec/support/spinabif.html http://sbanc.home.mindspring.com/ http://www.familyvillage.wisc.edu/lib_spin.htm www.evergreenspinabifida.org http://kimber.cjscreations.com/ http://www.spinabifida.org/ http://www.sbawny.org/Home.asp http://www.geocities.com/FetalSurgery_4_SB/ http://www.openroad.net.au/access/dakit/physical/phhandout4.htm http://www.sbaj.org/ http://www.icanride.org/bifida.html http://sbamn.com/ http://www.bifidaocculta.com/guestbook.cgi http://www.michaelclancy.com/ http://www.sbaco.org/ http://www.healthinsite.gov.au/topics/Spina_Bifida http://www.supportfind.com/category/spin02.asp http://www.ability.org.uk/Spina_Bifida.html http://www.cshen.org/resources/spinabifida.cfm http://brain.hastypastry.net/forums/forumdisplay.php?f⫽235 http://www.mic.ki.se/Diseases/C10.500.html http://spinabifidamomsogspot.com/ http://spinabifidaoccult.martoccult.com/ http://www.disabilityresources.org/SPINA.html http://www.kennedykrieger.org/kki_diag.jsp?pid⫽1105 http://dir.yahoo.com/Health/Diseases_and_Conditions/Spina_Bifida http://www.babyworld.co.uk/information/baby/special_needs/spina.asp http://www.bbc.co.uk/radio1/onelife/health/atoz/spina_bifida.shtml http://www.cafamily.org.uk/Direct/s42.html http://www.otdirect.co.uk/bifida.html http://www.spinalcord.ar.gov/Publications/FactSheets/sb.html http://wwww.dbpeds.org/articles/detail.cfm?TextID⫽107 http://www.va.gov/hac/forbeneficiaries/spina/spina.asp http://health.groups.yahoo.com/group/spinabifidacentral/ http://www.fpnotebook.com/NIC63.htm http://www.muhealth.org/⬃neuromed/tetheredcord.shtml (appendix continued)
1141
http://www.spinalcord.uab.edu/show.asp?durki⫽21760&site⫽ 1021&return⫽24203 http://www.ucsfhealth.org/childrens/medical_services/critical/sbifida/ http://www.amazon.com/gp/product/0933149603/103-9685110-0927056? v⫽glance&n⫽283155 http://www.clinicaltrials.gov/ct/gui/action/SearchAction?term⫽Spina⫹ Bifida http://abenews.go.com/Health/story?id⫽1456009 http://www.urbanext.uiuc.edu/specialneeds/spinebif.html http://www.cdc.gov/nchs/products/pubs/pubd/hestats/spine_anen.htm http://www.epill.com/spina.html ww2.kgw.com/Global/story.asp?S⫽1230739&nav⫽menu296_5 http://www.cnn.com/2005/HEALTH/conditions/12/28/iraq.spinabifida/ index.html http://www.legacyproductsinc.com/spina_bif.htm http://www.nemours.org/internet?url⫽no/ncc/svcs/div520.html http://www.injuryboard.com/view.cfm/Topic⫽602 http://www.sleeptight.com/EncyMaster/S/spina_bifida.html http://search.15000papers.com/cgi-bin/query?mss⫽15000papers&q⫽ Spina⫹Bifida http://www.infoplease.com/dictionary/spina⫹bifida http://sbawi.org/ http://www.shrinershq.org/patientedu/spinabifida.html http://spinabifidaconnection.com/forum/index.php http://ww3.komotv.com/Global/story.asp?S⫽2314420 http://www.dad.state.vt.us/dail/SpecificConditions/spinabifida.htm http://www.viaproject.org/spina.html http://www.cinn.org/ibsc/pediatric/spinabifida.html http://www.genomelink.org/spina/ http://www.emedicine.com/cgi-bin/foxweb.exe/searchengine@/em/ searchengine?boolean⫽and&book⫽all&maxhits⫽100&HiddenURL⫽ &query⫽Spina⫹Bifida
APPENDIX 2 10 High Quality Web Sites http://www.mayoclinic.com/health/spina-bifida/DS00417 http://www.sbhac.ca/index.php?page⫽main http://www.spinabifida.asn.au/ http://www.ucsfhealth.org/childrens/medical_services/critical/sbifida/ http://en.wikipedia.org/wiki/Spina_bifida http://checksutterfirst.org/spine/spina.html http://www.childrensmemorial.org/depts/motionanalysis/conditions/ spinabifida.asp http://www.emedicinehealth.com/articles/34669-1.asp http://www.ifglobal.org/home.asp?lang⫽1&main⫽1 http://www.ninds.nih.gov/disorders/spina_bifida/detail_spina_bifida.htm
REFERENCES 1.
Williams L J, Rasmussen SA, Flores A, Kirby RS and Edmonds LD: Decline in the prevalence of spina bifida and anencephaly by race/ethnicity: 1995–2002. Pediatrics 2005; 116: 580. 2. Honein MA, Paulozzi L J, Mathews TJ, Erickson JD and Wong LY: Impact of folic acid fortification of the US food supply on the occurrence of neural tube defects. JAMA 2001; 285: 2981. 3. Mitchell LE, Adzick NS, Melchionne J, Pasquariello PS, Sutton LN and Whitehead AS: Spina bifida. Lancet 2004; 364: 1885. 4. Zipitis CS and Paschalides C: Caring for a child with spina bifida: understanding the child and carer. J Child Health Care 2003; 7: 101. 5. Eysenbach G and Kohler C: What is the prevalence of healthrelated searches on the World Wide Web? Qualitative and quantitative analysis of search engine queries on the internet. AMIA Annu Symp Proc 2003; 225. 6. Eysenbach G, Powell J, Kuss O and Sa ER: Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002; 287: 2691. 7. Silberg WM, Lundberg GD and Musacchio RA: Assessing, controlling, and assuring the quality of medical information on the Internet: caveant lector et viewor—let the reader and viewer beware. JAMA 1997; 277: 1244.
1142 8.
9.
10.
11.
12.
13.
MYELOMENINGOCELE INFORMATION ON INTERNET
Sullivan D: Nielsen NetRatings search engine ratings. Available at http://searchenginewatch.com/reports/article.php/ 2156451. Accessed January 13, 2006. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS et al: Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001; 285: 2612. Oermann MH, Lowery NF and Thornley J: Evaluation of Web sites on management of pain in children. Pain Manag Nurs 2003; 4: 99. Paasche-Orlow MK, Taylor HA and Brancati FL: Readability standards for informed-consent forms as compared with actual readability. N Engl J Med 2003; 348: 721. American Public Health Association: Criteria for assessing the quality of health information on the Internet. Am J Public Health 2001; 91: 513. Wallace LS, Turner LW, Ballard JE, Keenum AJ and Weiss BD: Evaluation of web-based osteoporosis educational materials. J Womens Health (Larchmt) 2005; 14: 936.
14.
15.
16. 17.
18.
19.
20.
Mathur S, Shanti N, Brkaric M, Sood V, Kubeck J, Paulino C et al: Surfing for scoliosis: the quality of information available on the Internet. Spine 2005; 30: 2695. Maloney S, Ilic D and Green S: Accessibility, nature and quality of health information on the Internet: a survey on osteoarthritis. Rheumatology (Oxford) 2005; 44: 382. Black PC and Penson DF: Prostate cancer on the Internet— information or misinformation? J Urol 2006; 175: 1836. Zurmohle UM, Homann T, Schroeter C, Rothgerber H, Hommel G and Ermert JA: Psychosocial adjustment of children with spina bifida. J Child Neurol 1998; 13: 64. Gazmararian JA, Baker DW, Williams MV, Parker RM, Scott TL, Green DC et al: Health literacy among Medicare enrollees in a managed care organization. JAMA 1999; 281: 545. Pautler SE, Tan JK, Dugas GR, Pus N, Ferri M, Hardie WR et al: Use of the internet for self-education by patients with prostate cancer. Urology 2001; 57: 230. American Medical Association: JAMA and Archives. Available at http://pubs.ama-assn.org. Accessed January 8, 2006.
Purpose: We assessed the accessibility, readability and quality of myelomeningocele information on the Internet. Materials and Methods: We entered the term “spina bifida” into the Google, Yahoo! and Microsoft Network search engines, and stored the first 100 links from each Web site. A total of 164 unique Web sites remained for analysis, of which 159 were classified as relevant. Relevant Web sites were considered to have relevant content if more than 50% of the text was directly relevant to the disease, and to have relevant educational content if more than 50% of the text was aimed at relaying educational information. Readability was assessed using the Flesch-Kincaid Grade Level scale. The quality of the 159 Web sites with relevant content was assessed using American Public Health Association Criteria for Assessing Health Information on the Internet. Six criteria were assessed, namely credibility, content, links, design, interactivity and caveats. Web sites were analyzed by 2 separate medical doctors, with each reviewer blinded to the findings of the other. A weighted kappa statistic was used to calculate interrater reliability. Results: Of 159 relevant sites 146 (91.8%) had relevant content and 122 (76.7%) had relevant educational content. The average Flesch-Kincaid reading level was 10.9 (range 6 to 12). Quality was assessed on a 3-point scale, with 1 denoting poor quality, 2 fair and 3 good. Average scores were 1.92 for credibility, 1.88 for content, 2.29 for links, 2.53 for design, 1.99 for interactivity and 2.90 for caveats. The weighted kappa statistic for interrater reliability was 0.83. Conclusions: Myelomeningocele information on the Internet is relatively accessible, requires a high reading level for comprehension and is of variable quality. Key Words: meningomyelocele, spina bifida occulta, Internet, quality control
yelomeningocele currently affects approximately 70,000 people living in the United States.1 Before the United States Public Health Service folic acid mandate myelomeningocele affected approximately 1 in 1,600 pregnancies and 1 in 2,000 live births. Even after the mandate this condition is estimated to affect 1 in 2,440 pregnancies and 1 in 3,000 live births.1,2 Children with myelomeningocele have incomplete spinal cord development, with subsequent neurological deficits below the lesion. Associated impairments include brain malformations, hydrocephalus, neuropsychological difficulties, difficulties in bowel and bladder function, and difficulty in ambulation.3 While functional status with myelomeningocele varies among patients, coordination of care is complex and comprehension of the disease intricacies is a challenge for most families.4 It is estimated that up to 5% of all World Wide Web searches are health related, with up to 7 million health related searches conducted each day.5 Several reports have questioned the accuracy and readability of health related Web sites, and tools have been developed to study the quality, readability and suitability of health related sites.6,7 Internet information regarding myelomeningocele has yet to be studied. We assessed the accessibility, readability and quality of myelomeningocele information on the Internet.
M
Submitted for publication August 16, 2006.
0022-5347/07/1773-1138/0 THE JOURNAL OF UROLOGY® Copyright © 2007 by AMERICAN UROLOGICAL ASSOCIATION
MATERIALS AND METHODS Search engine selection was based on the Nielsen NetRatings, which showed that Google (46.2% of total online searches), Yahoo! (22.5%) and Microsoft Network (12.6%) were by far the most commonly used search engines.8 Between January 11 and January 13, 2006 trained searchers entered the term “spina bifida” into each of the 3 search engines, and the first 100 links from each Web site were stored. The total number of hits exceeded 200,000 on Google, 2,000,000 on Yahoo and 500,000 on Microsoft Network. Therefore, the first 100 links represented approximately 0.011% of the total number of hits but included the first sites to which patients would be linked when searching. This method of assessing multiple Web sites was consistent with previous studies.9 Web sites from academic, general health, personal, governmental, private profit, nonprofit and institutional sites were included, whereas advertisements linked to the results page were excluded. The figure outlines the process followed to categorize Web sites. Duplicate Web sites were excluded, leaving 164 unique Web sites for analysis. The links were then followed and classified as relevant if the search term was present in the text of the first electronic page to which the link led, resulting in 159 relevant Web sites. The 5 irrelevant Web sites were personal advertisement pages where the term “spina bifida” could be found in the text of 1 of the advertisements. Relevant Web sites were coded for content, and were considered to have relevant content if more than 50% of the
1138
Vol. 177, 1138-1142, March 2007 Printed in U.S.A. DOI:10.1016/j.juro.2006.10.019
MYELOMENINGOCELE INFORMATION ON INTERNET
1139
Web site classification process. MSN, Microsoft Network
text was directly relevant to the disease. In addition, Web sites with relevant content were classified as having relevant educational content if more than 50% of the text was aimed at relaying educational information to patients or families. The number of mouse clicks needed to reach the main educational page was recorded. The method of identifying accessible Web sites was consistent with previous studies.9,10 Readability was assessed using the Flesch-Kincaid Grade Level scale by copying all relevant text from the Web page and subsequent links into Microsoft Word, performing a spelling and grammar check, and recording the Flesch-Kincaid reading level.11 The quality of the 159 Web sites with relevant content was assessed using American Public Health Association Criteria for Assessing Health Information on the Internet.12 Six criteria were assessed, namely credibility, including the source, currency, relevance/utility, editorial review process for the information and financial disclosure; content, including accuracy, completeness and inclusion of appropriate disclaimers; links, including the relevance and comprehensiveness of links provided; design, including accessibility, navigability and internal search capacity; interactivity, including feedback mechanisms and means for exchange of information among users; and caveats, including clarification of whether the primary function of the site was to provide content or market products and services. Web sites were analyzed by 2 separate medical doctors (JB and RK), with each reviewer blinded to the findings of the other. A weighted kappa statistic was used to calculate interrater reliability. RESULTS Of 159 relevant sites 146 (91.8%) had relevant content and 122 (76.7%) had relevant educational content. Appendix 1 lists the 159 relevant Web sites. The average number of links needed to arrive at the main educational Web page was
0.42 (range 0 to 3), and the average Flesch-Kincaid reading level was 10.9 (6 to 12). Quality, including credibility, content, links, design, interactivity and caveats, was assessed on a 3-point scale, with 1 denoting poor quality, 2 fair and 3 good. The average scores were 1.92 for credibility, 1.88 for content, 2.29 for links, 2.53 for design, 1.99 for interactivity and 2.90 for caveats. The weighted kappa statistic for interrater reliability was 0.83. DISCUSSION Our findings addressed the accessibility, readability and quality of myelomeningocele information on the Internet. Accessibility findings were encouraging, with 159 of 164 Web sites (97%) having relevant information, of which 146 (91.8%) had relevant content and 122 (76.7%) had relevant educational content. These results are significantly better than those of previous accessibility studies performed for other medical problems.13–15 The number of links needed to arrive at educational information (0.42) suggests that myelomeningocele Web sites can be navigated with relative ease. Of the 6 quality indicators studied caveats rated most favorably, suggesting Web sites were clear in highlighting their primary function. Site design also rated well, suggesting that the accessibility, navigability and internal search capacity were fairly good. This finding reflected significant improvement over recent similar analyses for other diseases.4,6,7,9,13,15 Although the availability and relevance of links were above average, they surprisingly were inferior to design and caveats. Credibility, content and interactivity all rated below “fair.” Shortcomings in credibility were due to deficiencies in currency, relevance, utility and editorial review process. Black and Penson recently studied prostate cancer on the Internet and found similar deficiencies in credibil-
1140
MYELOMENINGOCELE INFORMATION ON INTERNET
ity, especially in currency of information.16 Deficiencies in content suggest the information patients and their families read on the Internet may be inaccurate or incomplete, or contain inappropriate disclaimers. Similar problems have been highlighted by others for manifold medical problems, although improvement with time has yet to be demonstrated.13–15 The failure of the Web sites to provide good interactivity is especially relevant for a disease such as myelomeningocele, where the lay public may be less informed than they would be regarding higher profile diseases such as prostate cancer or breast cancer. Children with myelomeningocele are at increased risk for poor psychosocial development, and interacting with peers at school and in other settings is vital to their psychosocial adjustment.17 Readability addresses the ability to comprehend the meaning of written text. The average adult in the United States reads at a 5th to 8th grade level, with wide variability.18 The Flesch-Kincaid Grade Level scale evaluates readability based on average sentence length and average number of syllables per word, assigning 3rd to 12th grade reading levels. This scale has been validated and used to assess readability for more than 40 years.11 Consistent with other studies, we found that the reading level of the Web sites reviewed (10.9) was significantly higher than the reading skill of most potential readers. Clearly, not all Web sites are created equal, and some are highly accessible and readable, and of superior quality. In 2001 Pautler et al19 recommended that national and international societies, including the American Urological Association, create a list of approved or endorsed Web sites, a recommendation that appears to be cogent today, as patients are increasingly turning to the Web for health information but with questionably improved efficiency. The American Medical Association has published recommendations for Web site design, although a mechanism needs to be developed to assess the degree to which sites abide by the recommendations.20 Ideally, Web sites would be easily accessible, with relevant educational content that is easily navigable with a minimal number of mouse clicks. They would have an internal search capacity as well as a mechanism for user feedback and interactivity among readers. Information would be current and appropriately referenced. Importantly, text would be written at no higher than an 8th grade reading level, consistent with the ability of the average reader. Links would be provided to academic, nonprofit, self-help and group help Web sites. Appendix 2 lists 10 of the highly ranked myelomeningocele Web sites. Our study has several limitations. The quality of information was subjectively assessed on a 3-point scale, although the weighted kappa statistic (0.83) would suggest the sites were reliably graded, with minimal interobserver variability. Web sites studied were chosen by their ranking on 3 separate search engines, not necessarily based on frequency of patient or family use (although this is 1 of the factors programmed into each of the 3 search engines). However, our analysis of 300 sites, of which 164 were unique, is more comprehensive than most studies performed for other diseases. Our study was further limited by the fact that the total number of Web sites analyzed represented only 0.011% of the total hits. However, the analysis included the top 100 hits from each of the 3 search engines, and, therefore, was
most relevant to the sites patients or families were likely to explore when searching for information. Future studies looking at physician knowledge of the Web sites patients are visiting would be interesting. Developing a ranking system, or index, would help direct patients toward useful and accurate Web sites. Studying whether the reading level of Web sites can be decreased without sacrificing the quality or accessibility of the information would also be useful. In conclusion, myelomeningocele information on the Internet is accessible, requires a high reading level for sufficient comprehension, is of variable quality and can be navigated with relative ease. APPENDIX 1 159 Relevant Web Sites http://www.geocities.com/HotSprings/2474/ http://www.iashah.ie/ http://nyneurosurgery.org/spinabifida_intro.htm http://www.illinoisspinabifidaassociation.com/ http://www.sbhac.ca/index.php?page⫽main http://www.asbha.org.au/ http://cpmenet.columbia.edu/dept/nsg/PNS/SpinaBifida.html http://www.sbaalbany.org/ http://www.dpo.uab.edu/⬃birmie/sb.htm http://rarediseases.about.com/cs/spinabifida/a/101902.htm http://www.emedicinehealth.com/articles/34669-1.asp http://www.ifglobal.org/home.asp?lang⫽1&main⫽1 http://en.wikipedia.org/wiki/Spina_bifida http://montekids.org/programs/spinabifida/ http://www.tvcc.on.ca/gateway.php?id⫽31&cid⫽2&PHPSESSID⫽ 594a36ad38aebb1580d212e4a492cf47 http://www.childrensmemorial.org/depts/motionanalysis/conditions/ spinabifida.asp http://www.webmd.com/hw/raising_a_family/hw169958.asp? lastselectedguid⫽{5FE84E90-BC77-4056-A91C-9531713CA348} http://www.ninds.nih.gov/disorders/spina_bifida/detail_spina_bifida.htm http://www.gillettechildrens.org/default.efm/PID⫽1.3.18 http://www2.vhihealthe.com/article/gale/100084077 http://www.cnn.com/HEALTH/library/DS/00417.html http://www.mydr.com.au/default.asp?article⫽3571 http://www.marchofdimes.com/pnhec/4439.asp http://www.mayoclinic.com/health/spina-bifida/DS00417 http://208.62.55.81/aboutus.ctm http://www.childrenshospital.org/az/Site1062/mainpageS1062P0.html http://www.chkd.org/Neurology/spibif.asp http://www.fetalcarecenter.org/medicine/fetal-surgery/myelomeningocelesurgery.htm http://kidshealth.org/parent/system/ill/spina_bifida.html http://www.asbah.org/ http://checksutterfirst.org/spine/spina.html http://health.yahoo.com/ency/healthwise/hw169956 http://www.emedicine.com/orthoped/topic557.htm http://www.findarticles.com/p/articles/mi_g2601/is_0012/ai_2601001279 http://www.healthatoz.com/healthatoz/Atoz/ency/spina_bifida.jsp http://groups.msn.com/PainIssuesForPersonsWithSpinaBifida http://www.fetal-surgery.com/ http://www.geocities.com/enchantedforest/cottage/1109/ http://www.spinabifida.asn.au/ http://www.sbatx.org/ http://www.sbhabc.org/ http://www.sbac.org/ http://spinabifida.mjsd.ca/info.htm http://www.chesapeakespinabifida.org/facts.htm http://www.juliasjourney.com/ http://www.spinabifidainfo.nl/ www.fortunecity.com/millenium/plumpton/268/sb.htm http://www.azspinabifida.org/ http://www.sbawa.asn.au/ http://sbahgc.org/faq/index.cfm http://www.spinabifidaofgeorgia.org/ http://www.sbahgc.org/ http://webess25.micromedex.com/content/DiseaseDex/001258.htm http://www.cynsspot.com/SpinaBifida.htm http://www.spinabifidamoms.com/english/about.html http://www.sbhao.on.ca/ http://ibis-birthdefects.org/start/ntdfact.htm (appendix continued)
MYELOMENINGOCELE INFORMATION ON INTERNET APPENDIX 1 continued
APPENDIX 1 continued
159 Relevant Web Sites
159 Relevant Web Sites
http://www.keepkidshealthy.com/welcome/conditions/spinabifida.html http://www.medicinenet.com/spina_bifida_and_anencephaly/article.htm http://faculty.washington.edu/chudler/spibi.html http://www.specialchild.com/archives/dz-005.html http://www.ssba.org.uk/ http://www.spineuniverse.com/displayarticle.php/article234.html www.drgreene.com/21_1191.html http://www.msbaweb.org/ http://www.uromed.com/?kwov⫽children_with_spina_bifida&OVRAW⫽ spina%20bifida&OVKEY⫽child%20with%20spina%20bifida&OVMTC⫽ advanced http://www.nichcy.org/pubs/factshe/fs12txt.htm http://www.bbe.co.uk/health/conditions/spinabifida2.shtml http://www.intelihealth.com/IH/ihtIH/WSIHW000/9339/9619.html http://stephblog.com/?cat⫽17 http://members.tripod.com/⬃imaware/sb.html http://www.hopkinsmedicine.org/orthopedicsurgery/spinabifida.html http://ymghealthinfo.org/content.asp?page⫽P02624 http://kidshealth.org/kid/health_problems/brain/spina_bifida.html http://www.irishhealth.com/?level⫽4&con⫽471 http://www.drgreene.com/21_1191.html http://www.neurosurgerytoday.org/what/patient_e/spina.asp http://ww2.kgw.com/Global/story.asp?S⫽1230739&nav⫽menu296_5 http://www.netwellness.org/ency/article/001558.htm http://www.kidshealth.org/kid/health_problems/bone/spina_bifida.html http://www.nlm.nih.gov/medlineplus/ency/article/001558.htm http://www.hmc.psu.edu/healthinfo/s/spinabifida.htm http://www.ivillage.co.uk/pregnancyandbaby/baby/babyhealth/articles/ 0,30_163464,00.html http://www.geocities.com/SBAofgpa/ http://www.geocities.com/Heartland/Acres/1297/spinabifida.html http://www.eplibrary.com/spinabifida/ http://www.exceptionalteaching.com/bifida.html http://www3.ns.sympatico.ca/spina.bifida/ http://www.srhsb.org/ http://latexallergylinks.tripod.com/sb.html http://users.iconz.co.nz/apeacock/mobility.html http://www.sbak.org/ http://www.sbfv.org.au/ http://www.sbatsr.org/ http://www.sbawp.org/ http://www.spinabifidalaw.com/ http://www.sbagno.org/ http://www.waisman.wise.edu/⬃rowley/sb_kids.htmlx http://www.kumc.edu/gec/support/spinabif.html http://sbanc.home.mindspring.com/ http://www.familyvillage.wisc.edu/lib_spin.htm www.evergreenspinabifida.org http://kimber.cjscreations.com/ http://www.spinabifida.org/ http://www.sbawny.org/Home.asp http://www.geocities.com/FetalSurgery_4_SB/ http://www.openroad.net.au/access/dakit/physical/phhandout4.htm http://www.sbaj.org/ http://www.icanride.org/bifida.html http://sbamn.com/ http://www.bifidaocculta.com/guestbook.cgi http://www.michaelclancy.com/ http://www.sbaco.org/ http://www.healthinsite.gov.au/topics/Spina_Bifida http://www.supportfind.com/category/spin02.asp http://www.ability.org.uk/Spina_Bifida.html http://www.cshen.org/resources/spinabifida.cfm http://brain.hastypastry.net/forums/forumdisplay.php?f⫽235 http://www.mic.ki.se/Diseases/C10.500.html http://spinabifidamomsogspot.com/ http://spinabifidaoccult.martoccult.com/ http://www.disabilityresources.org/SPINA.html http://www.kennedykrieger.org/kki_diag.jsp?pid⫽1105 http://dir.yahoo.com/Health/Diseases_and_Conditions/Spina_Bifida http://www.babyworld.co.uk/information/baby/special_needs/spina.asp http://www.bbc.co.uk/radio1/onelife/health/atoz/spina_bifida.shtml http://www.cafamily.org.uk/Direct/s42.html http://www.otdirect.co.uk/bifida.html http://www.spinalcord.ar.gov/Publications/FactSheets/sb.html http://wwww.dbpeds.org/articles/detail.cfm?TextID⫽107 http://www.va.gov/hac/forbeneficiaries/spina/spina.asp http://health.groups.yahoo.com/group/spinabifidacentral/ http://www.fpnotebook.com/NIC63.htm http://www.muhealth.org/⬃neuromed/tetheredcord.shtml (appendix continued)
1141
http://www.spinalcord.uab.edu/show.asp?durki⫽21760&site⫽ 1021&return⫽24203 http://www.ucsfhealth.org/childrens/medical_services/critical/sbifida/ http://www.amazon.com/gp/product/0933149603/103-9685110-0927056? v⫽glance&n⫽283155 http://www.clinicaltrials.gov/ct/gui/action/SearchAction?term⫽Spina⫹ Bifida http://abenews.go.com/Health/story?id⫽1456009 http://www.urbanext.uiuc.edu/specialneeds/spinebif.html http://www.cdc.gov/nchs/products/pubs/pubd/hestats/spine_anen.htm http://www.epill.com/spina.html ww2.kgw.com/Global/story.asp?S⫽1230739&nav⫽menu296_5 http://www.cnn.com/2005/HEALTH/conditions/12/28/iraq.spinabifida/ index.html http://www.legacyproductsinc.com/spina_bif.htm http://www.nemours.org/internet?url⫽no/ncc/svcs/div520.html http://www.injuryboard.com/view.cfm/Topic⫽602 http://www.sleeptight.com/EncyMaster/S/spina_bifida.html http://search.15000papers.com/cgi-bin/query?mss⫽15000papers&q⫽ Spina⫹Bifida http://www.infoplease.com/dictionary/spina⫹bifida http://sbawi.org/ http://www.shrinershq.org/patientedu/spinabifida.html http://spinabifidaconnection.com/forum/index.php http://ww3.komotv.com/Global/story.asp?S⫽2314420 http://www.dad.state.vt.us/dail/SpecificConditions/spinabifida.htm http://www.viaproject.org/spina.html http://www.cinn.org/ibsc/pediatric/spinabifida.html http://www.genomelink.org/spina/ http://www.emedicine.com/cgi-bin/foxweb.exe/searchengine@/em/ searchengine?boolean⫽and&book⫽all&maxhits⫽100&HiddenURL⫽ &query⫽Spina⫹Bifida
APPENDIX 2 10 High Quality Web Sites http://www.mayoclinic.com/health/spina-bifida/DS00417 http://www.sbhac.ca/index.php?page⫽main http://www.spinabifida.asn.au/ http://www.ucsfhealth.org/childrens/medical_services/critical/sbifida/ http://en.wikipedia.org/wiki/Spina_bifida http://checksutterfirst.org/spine/spina.html http://www.childrensmemorial.org/depts/motionanalysis/conditions/ spinabifida.asp http://www.emedicinehealth.com/articles/34669-1.asp http://www.ifglobal.org/home.asp?lang⫽1&main⫽1 http://www.ninds.nih.gov/disorders/spina_bifida/detail_spina_bifida.htm
REFERENCES 1.
Williams L J, Rasmussen SA, Flores A, Kirby RS and Edmonds LD: Decline in the prevalence of spina bifida and anencephaly by race/ethnicity: 1995–2002. Pediatrics 2005; 116: 580. 2. Honein MA, Paulozzi L J, Mathews TJ, Erickson JD and Wong LY: Impact of folic acid fortification of the US food supply on the occurrence of neural tube defects. JAMA 2001; 285: 2981. 3. Mitchell LE, Adzick NS, Melchionne J, Pasquariello PS, Sutton LN and Whitehead AS: Spina bifida. Lancet 2004; 364: 1885. 4. Zipitis CS and Paschalides C: Caring for a child with spina bifida: understanding the child and carer. J Child Health Care 2003; 7: 101. 5. Eysenbach G and Kohler C: What is the prevalence of healthrelated searches on the World Wide Web? Qualitative and quantitative analysis of search engine queries on the internet. AMIA Annu Symp Proc 2003; 225. 6. Eysenbach G, Powell J, Kuss O and Sa ER: Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002; 287: 2691. 7. Silberg WM, Lundberg GD and Musacchio RA: Assessing, controlling, and assuring the quality of medical information on the Internet: caveant lector et viewor—let the reader and viewer beware. JAMA 1997; 277: 1244.
1142 8.
9.
10.
11.
12.
13.
MYELOMENINGOCELE INFORMATION ON INTERNET
Sullivan D: Nielsen NetRatings search engine ratings. Available at http://searchenginewatch.com/reports/article.php/ 2156451. Accessed January 13, 2006. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS et al: Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001; 285: 2612. Oermann MH, Lowery NF and Thornley J: Evaluation of Web sites on management of pain in children. Pain Manag Nurs 2003; 4: 99. Paasche-Orlow MK, Taylor HA and Brancati FL: Readability standards for informed-consent forms as compared with actual readability. N Engl J Med 2003; 348: 721. American Public Health Association: Criteria for assessing the quality of health information on the Internet. Am J Public Health 2001; 91: 513. Wallace LS, Turner LW, Ballard JE, Keenum AJ and Weiss BD: Evaluation of web-based osteoporosis educational materials. J Womens Health (Larchmt) 2005; 14: 936.
14.
15.
16. 17.
18.
19.
20.
Mathur S, Shanti N, Brkaric M, Sood V, Kubeck J, Paulino C et al: Surfing for scoliosis: the quality of information available on the Internet. Spine 2005; 30: 2695. Maloney S, Ilic D and Green S: Accessibility, nature and quality of health information on the Internet: a survey on osteoarthritis. Rheumatology (Oxford) 2005; 44: 382. Black PC and Penson DF: Prostate cancer on the Internet— information or misinformation? J Urol 2006; 175: 1836. Zurmohle UM, Homann T, Schroeter C, Rothgerber H, Hommel G and Ermert JA: Psychosocial adjustment of children with spina bifida. J Child Neurol 1998; 13: 64. Gazmararian JA, Baker DW, Williams MV, Parker RM, Scott TL, Green DC et al: Health literacy among Medicare enrollees in a managed care organization. JAMA 1999; 281: 545. Pautler SE, Tan JK, Dugas GR, Pus N, Ferri M, Hardie WR et al: Use of the internet for self-education by patients with prostate cancer. Urology 2001; 57: 230. American Medical Association: JAMA and Archives. Available at http://pubs.ama-assn.org. Accessed January 8, 2006.