- Email: [email protected]
Quality of fertility clinic websites
CONTROVERSY: WEBSITE: QUALITY ASSURED? Quality of fertility clinic websites Jack Y. J. Huang, M.D., Federico Discepola, M.D., Haya Al-Fozan, M.D., and Togas Tulandi, M.D. Department of Obstetrics and Gynecology, McGill University, Montreal, Que´bec, Canada
Objective: To evaluate the overall quality of the Society for Assisted Reproductive Technology (SART)-affiliated fertility clinic websites, according to American Medical Association (AMA) Internet health information guidelines. Design: Cross-sectional evaluation. Setting: Two hundred sixty-six websites drawn from the SART directory and the World Wide Web. Intervention(s): Three objective scoring scales (ownership, content, and navigation) adapted from AMA guidelines. Main Outcome Measures: Seven objective criteria for ownership, 8 for content, and 11 for website navigation. Result(s): Two thirds of SART-affiliated fertility clinics have functional websites. Of the 236 sites evaluated, 58 belong to hospital centers. Overall, the scores for the three scoring scales were low. Compared with the websites of non-hospital clinics, those of hospital centers were more likely to include information about site ownership and affiliations (89.7% vs. 60.7%) and patient privacy (34.5% vs. 20.8%). Also, contents of hospital center websites were significantly easier to distinguish from advertisements (70.7% vs. 47.7%), and reference sources for specific contents were easier to identify (27.6% vs. 8.4%). Hospital center websites were more likely to indicate affiliations and financial disclosures of authors (25.9% vs. 10.7%), to feature a site map (25.9% vs. 12.3%), and to have a search function (31.0% vs. 5.6%). Conclusion(s): Websites of SART-affiliated clinics fail to meet most of the AMA health information guidelines. The quality of the hospital centers’ websites is better than that of private clinics. (Fertil Steril威 2005;83:538 – 44. ©2005 by American Society for Reproductive Medicine.) Key Words: Internet, World Wide Web, fertility, Society for Assisted Reproductive Technology (SART)
In recent years, the Internet has become more accessible to the general public, providing quick and convenient access to health information. Health-related websites represent one of the most common noncommercial web applications in the world (1). A vast amount of medical information is available through an estimated 20,000 to 100,000 health-related websites (2). In fact, the field of reproductive medicine is extensively represented on the World Wide Web. A recent search of the keyword “infertility” with the popular search-engine Google revealed 1,840,000 web pages (3). An estimated 52 million Americans browse the Internet to access health or medical information (4). Furthermore, studies done in Canada and The Netherlands showed that nearly half of infertile couples consult the Internet for fertilityrelated issues (5–7). We recently published a study looking at the reasons for Internet use and the source of online reproductive health information accessed by infertile couples at our institution (5). We found that 60% of women and 45% of men browse the Internet to obtain information on medical Received March 8, 2004; revised and accepted August 12, 2004. Reprint requests: Togas Tulandi, M.D., Department of Obstetrics and Gynecology, McGill University, 687 Pine Avenue West, Montreal, Que´bec H3A 1A1, Canada (FAX: 514-843-1448; E-mail: [email protected]).
538
conditions. In addition, 79% of men and 48% of women use the Internet to search for a second opinion from other fertility centers. More importantly, 62% of women and 41% of men stated that their decision to seek fertility treatments was influenced by health information on the Internet. As sources for online health information, academic institution or university websites (30.3%) and fertility clinic websites (28.1%) were the three most commonly cited sources (5). A popular reference is the Society for Assisted Reproductive Technology (SART) website, which provides a listing of 357 affiliated fertility clinics, including their contact information and website addresses (8). Unfortunately, the information provided on the Internet is often incomplete, misleading, or inaccurate (9). More specifically, the quality of online health information varies and might be biased by commercial interests. Furthermore, by accessing health information online, there is a risk of involuntarily revealing personal information or patterns of use and interests to third parties. To address these issues, the American Medical Association (AMA) published guidelines for medical and health information websites (10) to guide the development of web contents, to govern online advertising
Fertility and Sterility姞 Vol. 83, No. 3, March 2005 Copyright ©2005 American Society for Reproductive Medicine, Published by Elsevier Inc.
0015-0282/05/$30.00 doi:10.1016/j.fertnstert.2004.08.036
and sponsorships, and to protect the privacy and confidentiality of site visitors and patients. Currently, little is known about the quality of the fertility clinic websites in the United States. The purpose of our study was to evaluate the quality of the online health information from the SART-affiliated fertility clinics, using the AMA guidelines. MATERIALS AND METHODS A list of websites for SART-affiliated fertility clinics was obtained from the SART website (8). If a clinic’s website address was not listed in the SART directory, we used two search engines, Google (11) and Yahoo! (12), to search for the name of the fertility clinic and the medical director. Only clinics with websites were reviewed. Two coauthors (J.Y.J.H. and F.D.) evaluated each website. The fertility clinic websites were classified according to the practice settings (hospital-sponsored centers or independently operated clinics) and the site affiliations (universities, other health information websites, and pharmaceutical companies). A website was considered to be hospital sponsored if the affiliation was clearly stated in the web pages. Evaluation Criteria The quality of online health information was assessed according to three objective scoring scales adapted from the AMA online health information guidelines (ownership and affiliation, editorial content, and navigation of web contents) (10). We developed seven objective criteria to assess information regarding website ownership and affiliations, with a score of 1 given for each of the criteria met, for a maximum score of 7: 1. Website ownership, including affiliations, strategic alliances, and significant investors, should be clearly indicated on the home screen or via a directly accessible link on the home screen. 2. Copyright ownership of specific content should be clearly indicated. 3. Information about restrictions on access to content, required registration, and password protection, if applicable, should be provided and easy to find. 4. Information about payment subscriptions, document delivery, and viewing should be provided and easy to find. 5. Information about privacy should be provided and easy to find. 6. Funding or other sponsorship for any specific content should be clearly indicated. 7. Content should be easily distinguished from advertising. Editorial content was assessed according to eight objective criteria, with a score of 1 given for each of the criteria met, for a maximum score of 8: Fertility and Sterility姞
1. Content should be reviewed for quality (including originality, accuracy, and reliability) by either peer review or editorial boards. 2. The language complexity of the content should be appropriate for the site’s audience. 3. A description of the editorial process and method of content review should be posted on the site. 4. All staff members and other individuals (i.e., editorial board) responsible for content quality should be posted on site. 5. The dates on which content is posted, revised, and updated should be clearly indicated. 6. Sources for specific content should be clearly identified (i.e., author byline or names of individual, organization, department, institution, agency, or commercial provider/ producer). 7. Affiliations and relevant financial disclosures for authors and content producers should be clearly indicated. 8. Reference material used to develop content should be cited in a manner appropriate for the site’s audience. Navigation of web contents was evaluated according to 11 specific criteria, with a score of 1 given for each of the criteria met, for a maximum score of 11: 1. The website should provide information about the platform(s) and browser (that permit optimal viewing in a location that is easy to find). 2. The intrasite content links should be functional. 3. The external links should be functional. 4. The website allows viewers to return to previously browsed websites. 5. The website does not redirect viewer to unintended websites. 6. Information on portable document format (PDF) files and software download instruction should be provided and easy to find. 7. The website features include a site map. 8. The website features include a “frequently asked questions” (FAQ) page. 9. The website features include a feedback mechanism. 10. The website features include customer service information. 11. Each distinct site should provide a search engine with instructions specifying how to use the search function. The two evaluators tested all the intrasite and external hyperlinks of each website. All links had to be functional to be awarded 1 point. Partial credits were not awarded. Statistical Analysis For comparison between hospital-sponsored centers and private clinics, and between university-affiliated clinics and non– university-affiliated clinics, we used the 2 test for categoric variables and the Mann-Whitney test for continuous variables. Results are expressed as mean and 95% confidence interval; the differences were considered statistically significant if P⬍.05. 539
TABLE 1 Website ownership information.
All websites Hospitals (n ⴝ 236) (n ⴝ 58) Website ownership, affiliations, strategic alliances, and significant investors Copyright ownership Restrictions of access to content, required registration, and password protection Payment subscription, document delivery, and viewing Privacy protection Funding or sponsorship information Content is easily distinguished from advertising
160 (67.8)
Clinics (n ⴝ 178)
52a (89.6) 108a (60.7)
Universityaffiliated centers (n ⴝ 52)
Non–universityaffiliated centers (n ⴝ 184)
46a (88.5)
114a (61.9)
91 (38.5) 17 (7.2)
19 (32.7) 5 (8.6)
73 (41.0) 12 (6.7)
18 (34.6) 3 (5.7)
74 (40.2) 14 (7.6)
44 (18.6)
7 (12.1)
37 (20.8)
6 (11.5)
38 (20.6)
56 (23.7) 11 (4.7)
20b (34.5) 5 (8.6)
37b (20.8) 6 (3.4)
15 (28.8) 5 (9.6)
42 (22.8) 6 (3.3)
126 (53.4)
41b (70.7)
85b (47.7)
37b (71.1)
89b (48.4)
Note: Data are presented as n (%). a P⬍.001. b P⬍.05. Huang. Quality of fertility clinic websites. Fertil Steril 2005.
RESULTS Two thirds (66.1%) of SART-affiliated fertility clinics in the United States had functional websites. Among the 236 websites evaluated, 58 centers were sponsored by hospitals and 178 by independently operated fertility clinics. Twenty-two percent of the websites were affiliated with academic institutions, and 20.3% of them were affiliated with other health information websites.
reliability by either peer review or editorial boards. Only 2.5% of websites provided a description of the editorial process. Just 13.1% of websites identified the source of specific contents, and 16.5% of websites cited reference materials in a manner appropriate for the site’s audience. As for the timeliness of the web contents, only 11.4% of websites posted the dates of the content publications and revisions.
Overall, the scores for the three scoring scales were low. The mean score for the 7-point assessment of site ownership was 2.1. The mean score for the 8-point assessment of quality of website content was 2.1. The mean score for the 11-point evaluation of website navigation functions was 6.0.
Regarding the navigational functionality of the websites, most websites featured functional intrasite and external links, allowed viewers to return to a previously browsed web page, and did not redirect viewers to unintended websites (Table 3). Customer feedback mechanisms existed in 93.2% of websites. However, most websites lacked functions such as a site map, a FAQ page, a search engine, and download instructions for PDF files.
Regarding website ownership information, 67.8% of websites clearly indicated information about ownership, including affiliations, strategic alliances, and significant investors (Table 1). However, only 38.5% of websites indicated copyright information of specific web contents. More importantly, only 23.7% of websites included information about patient privacy policies, and a mere 4.7% of websites revealed funding or other sponsorship for any specific content. Regarding the quality of the website content, 50% of websites listed all staff members and individuals responsible for content quality (Table 2). However, only 3.4% of the websites evaluated attested that their web contents were reviewed for quality, including originality, accuracy, and 540
Huang et al.
Quality of fertility clinic websites
Comparisons Between Hospital-Sponsored Centers and Private Clinics Compared with private clinics, hospital-sponsored center websites were more likely to include information about ownership and affiliation (89.7% vs. 60.7%, P⬍.001) and patient privacy (34.5% vs. 20.8%, P⬍.05). Contents of the hospital center websites were significantly easier to distinguish from advertisements (70.7% vs. 47.8%, P⬍.05). Reference sources for specific contents were significantly easier to identify in hospital center websites (27.6% vs. 8.4%, Vol. 83, No. 3, March 2005
TABLE 2 Website content.
Content is reviewed by editorial boards Appropriate language complexity Editorial process description Individuals responsible for content are posted on site The content is revised and updated regularly Source of content is identified Affiliations and relevant financial disclosures for authors Reference material is cited
All websites (n ⴝ 236)
Hospitals (n ⴝ 58)
Clinics (n ⴝ 178)
Universityaffiliated centers (n ⴝ 52)
Non–universityaffiliated centers (n ⴝ 184)
8 (3.4)
1 (1.7)
7 (3.9)
2 (3.8)
6 (3.3)
233 (98.7)
58 (100)
176 (98.8)
52 (100)
182 (98.9)
6 (2.5) 118 (50.0)
0 (0) 25 (43.1)
6 (3.4) 94 (52.8)
0 (0) 25 (48.1)
6 (3.3) 94 (51.1)
27 (11.4)
10 (17.2)
17 (9.6)
9 (17.3)
18 (9.8)
31 (13.1) 34 (14.4)
16a (27.6) 15b (25.9)
15a (8.4) 19b (10.7)
15a (28.8) 17a (32.7)
16a (8.7) 17a (9.2)
39 (16.5)
13 (22.4)
26 (14.6)
13 (25)
26 (14.1)
Note: Data are presented as n (%). a P⬍.001. b P⬍.05. Huang. Quality of fertility clinic websites. Fertil Steril 2005.
P⬍.001). Also, hospital center websites were more likely to indicate affiliations and financial disclosures of authors (25.9% vs. 10.7%, P⬍.05), to feature a site map (25.9% vs. 12.4%, P⬍.05), and to have a search function (31.0% vs. 5.6%, P⬍.001). Comparison Between University-Affiliated Clinics and Non–University-Affiliated Clinics University-affiliated centers were more likely to include ownership and affiliation information (88.5% vs. 62.0%, P⬍.001) than non-university centers. Web contents were significantly easier to distinguish from advertisements (71.2% vs. 48.4%, P⬍.05), and reference sources were significantly easier to identify (28.8% vs. 8.7%, P⬍.001). Also, university-affiliated clinic websites were more likely to indicate relevant financial disclosures of authors (32.7% vs. 9.2%, P⬍.001) and to include a search function (26.9% vs. 7.6%, P⬍.001). DISCUSSION This is the first study to evaluate the quality of the online health information from SART-affiliated fertility clinics, according to objective criteria based on the AMA guidelines. We found that two thirds of the SART-affiliated fertility clinics have functional websites. Irrespective of the practice setting or university affiliation, the overall quality of the fertility clinic websites is poor, failing to meet most of the Fertility and Sterility姞
AMA Internet health information guidelines. Most websites failed to provide information about copyright or sponsorship of any specific content, payment subscriptions or methods of document delivery and viewing, and the protection of viewer privacy. Regarding the quality of website content, most sites lacked descriptions of the editorial process, the source of specific contents, the proper citations of reference sources, the dates on which the contents were posted, revised, and updated, and relevant financial disclosures of authors and content producers. Although most websites featured functional links, the majority of websites lacked features such as a site map, a FAQ page, and search functions. Moreover, most websites did not provide clear information about the web browser recommended for optimal viewing or instructions for downloading PDF document files. Owing to the lack of a peer-review process, such as the one used for medical journals, the validity of the information on the Internet is sometimes questionable. Some of these health-related sites are created and maintained by academic institutions, some by healthcare groups (2), and others by commercial companies with the intent to advertise a product. A large part of the material is helpful, but one often finds incorrect or incomplete information (13). Furthermore, the language complexity and the clarity of the information are not standardized and might not be suitable for individuals without proper medical backgrounds (14). 541
TABLE 3 Website navigation. University-affiliated All websites Hospitals Clinics centers (n ⴝ 236) (n ⴝ 58) (n ⴝ 178) (n ⴝ 52) The site provides information about the browser Functional intrasite links Functional external links Viewers allowed to return to previous websites The website does not redirect viewer to unintended websites Portable document format file (PDF) download instructions are provided and easy to find Features a site map Features “frequently asked questions” page Features feedback mechanism Features customer service information Provides a search engine
5 (2.1)
0 (0)
5 (2.7)
0 (0)
Non–universityaffiliated centers (n ⴝ 184) 5 (2.7)
227 (96.2) 229 (97.0) 206 (87.3)
57 (98.3) 171 (96.1) 57 (98.3) 173 (97.2) 51 (87.9) 156 (87.6)
51 (98.1) 51 (98.1) 45 (86.5)
177 (96.2) 179 (97.3) 162 (88.0)
227 (96.2)
56 (96.5) 172 (96.6)
50 (96.1)
178 (96.7)
20 (11.2)
7 (13.5)
20 (10.9)
36 (15.2) 77 (32.6)
15b (25.9) 22b (12.3) 17 (29.3) 61 (34.3)
12 (23.1) 17 (32.7)
25 (13.6) 61 (33.1)
220 (93.2)
55 (94.8) 166 (93.2)
50 (96.1)
171 (92.9)
119 (50.4)
32 (55.2)
27 (51.9)
92 (50)
14a (26.9)
14a (7.6)
27 (11.4)
28 (11.9)
7 (12.1)
87 (48.9)
18a (31.0) 10a (5.6)
Note: Data are presented as n (%). a P⬍.001. b P⬍.05. Huang. Quality of fertility clinic websites. Fertil Steril 2005.
In 1999, the American Society for Reproductive Medicine published guidelines for advertising by assisted reproductive technology (ART) programs. However, the guidelines are limited to the advertisement of successful pregnancy rates from ART (15). Several other health organizations have developed generic rating tools to objectively evaluate online health information, including the Health On the Net (HON) Foundation and the British Healthcare Internet Association (16, 17). These guidelines are similar to the Journal of the American Medical Association’s minimal core standards for responsible printed documents, including authorship of the content, attribution to the sources of the content, disclosure of funding and competing interests, and timeliness of the information presented (9). Unfortunately, we discovered that even websites that displayed the HON insignia did not always comply with all AMA guidelines. This finding is even more disturbing because the HON insignia implies that the website conforms to its quality assurance criteria. This is an alarming trend, which has been noted in other healthcare literature. For example, a study evaluating the quality of 542
Huang et al.
Quality of fertility clinic websites
online asthma information also reported websites with the HON insignia that did not conform to all the HON guidelines (18). We compared websites sponsored by hospital centers and by independently operated clinics to evaluate one of our research hypotheses: the quality of hospital center websites is superior to that of private clinics. Hospitals often have more elaborate websites and might have stricter guidelines for publication of health information. Similarly, we believe that websites affiliated with academic institutions are more extensive in content and frequently include links to health research and educational websites. Findings from this study are similar to those from other studies evaluating the quality of health information on the Internet. Okamura et al. (19) evaluated 197 fertility-related websites, using the AMA’s four minimal core standards for responsible printed documents. The investigators found that only 2% of websites met all four recommended standards, and up to 50% of websites failed to report any of the four Vol. 83, No. 3, March 2005
standards. They concluded that commercial websites were more likely than educational websites to fail to meet minimum standards. In contrast, we did not find any difference between the quality of academic institution–affiliated clinic websites and private clinic websites. Nonetheless, the quality of Internet resources available to infertile couples is at best variable. A survey of online health users cited the protection of personal privacy as the most important priority (20). Current technology allows the host site or third parties to collect personal information over the Internet and to track health information–seeking behaviors, which could breach personal privacy and reveal the user’s health data. In the present study, we found that only 23.7% of websites feature patient privacy protection statements. This is an alarming finding because the percentage might be actually an overestimation. It has been shown in previous studies that although many health information websites state privacy policies, most of the websites do not follow them (21). There are some limitations to our study. It was based on American websites listed on the SART website (8), and not on websites listed according to search engines. It is possible that the 236 websites surveyed were not among the most popular websites visited by fertility couples. Moreover, some clinics might have websites that were not accessible with our search methods. We did not actually contact the clinics and ask whether they have functional websites. However, we believe that our method of using websites recognized by SART is reasonable because there are few reliable and complete lists of fertility clinic websites. Furthermore, SART includes more than 370 practicing members, representing more than 95% of the ART clinics in the nation. Second, the World Wide Web is a constantly evolving technology, and the information available on the Internet is constantly being altered. In any event, we believe that our study reflects the most current state of the technology and practice on the World Wide Web. Although numerous health organizations have published instruments and guidelines to evaluate online health information, the reliability and validity of these tools have been questioned (16, 22–24). Our purpose was to evaluate information from SART-affiliated websites according to a set of objective scoring criteria. We chose the AMA Guidelines for Medical and Health Information Sites over other rating tools because it has been adopted by the Health Summit Working Group and has been used in numerous studies (25). Last, the interrater reliability was not assessed. Nevertheless, because the websites were evaluated according to a set of objective criteria, we believe that interrater reliability should not be a significant factor in this study. The widespread use of the Internet by patients poses a modern challenge to obstetricians and gynecologists. Patients are taking a more active role in deciding about their health care (26) and have realized that information from doctors is often inadequate to satisfy their desire for knowlFertility and Sterility姞
edge regarding health (5, 27). Furthermore, 21% of Internet users prefer being followed by physicians with their own websites (9). Physicians must anticipate that patients might enter their practices with information concerning their medical conditions or treatments obtained from the Internet. A standard patient visit should therefore include questions regarding what the patients are learning from their Internet searches for health information and educating patients on how to better evaluate quality of information on the Internet. Physicians must assume a new role of health information provider, maintaining their own health websites with accurate and current information or directing their patients to reliable and up-to-date Internet health websites. More importantly, professional organizations governing the practice of ART should develop more specific guidelines on online health information to ensure that patients have access to the most reliable, current, and high-quality health information on reproductive medicine. REFERENCES 1. Lawrence S, Giles CL. Accessibility of information on the Web. Nature 1999;400:107–9. 2. Eysenbach G, Sa ER, Diepgen TL. Shopping around the Internet today and tomorrow: towards the millennium of cybermedicine. BMJ 1999; 319:1294 – 8. 3. Google. Search results for term “infertility.” Available at: www.google. com. Accessed July 21, 2004. 4. Fox S, Rainee L, Horrigan J, Lenhart A, Spooner T, Burke M, et al. The online healthcare revolution: how the Web helps Americans take better care of themselves. Washington, DC: Pew Internet and American Life Project. Available at: http://www.pewinternet.org. Accessed July 21, 2004. 5. Huang JYJ, Al-Fozan H, Tan SL, Tulandi T. Internet use by patients seeking fertility treatment. Int J Gynecol Obstet 2003;83:75– 6. 6. Haagen EC, Tuil W, de Bruijn RP, Braat DDM, Kremer JAM. Current Internet use and preferences of IVF and ICSI patients. Hum Rep 2003;18:2073– 8. 7. Weissman A, Gotlieb L, Ward S, Greenblatt E, Casper R. Use of Internet by infertile couples. Fertil Steril 2000;73:1180 –2. 8. Society for Assisted Reproductive Technology. Available at: www. sart.org. Accessed April 1–15, 2003. 9. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling and assuring the quality of medical information on the Internet: caveant lector et viewor—let the reader and viewer beware. JAMA 1997;277: 1244 –5. 10. Winker MA, Flanagin A, Chi-Lum B, White J, Andrews K, Kennett RL, et al. Guidelines for medical and health information sites on the Internet. JAMA 2000;283:1600 – 6. 11. Google. Available at: www.google.com. Accessed April 1–15, 2003. 12. Yahoo! Available at: http://www.yahoo.com. Accessed April 1–15, 2003. 13. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the World Wide Web: systematic survey of advice on managing fever in children at home. BMJ 1997;314: 1875–9. 14. Graber MA, Roller CM, Kaeble B. Readability levels of patient education material on the World Wide Web. J Fam Pract 1999;48:58 – 61. 15. American Society for Reproductive Medicine. A practice committee report. Guidelines for advertising by ART programs 1999. Available at: http://www.asrm.org/membersonly/practice/ArtAdvertising.pdf. Accessed July 21, 2004. 16. Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related websites: review. BMJ 1999;318:647–9. 17. Health on the Net Foundation. Health on the Net code of conduct. Avail-
543
18. 19.
20.
21.
able at: http://www.hon.ch/honcode/honcode_check.html. Accessed July 21, 2004. Croft DR, Peterson MW. An evaluation of the quality and contents of asthma education on the World Wide Web. Chest 2002;121:1301–7. Okamura K, Bernstein J, Fidler AT. Assessing the quality of fertility resources on the World Wide Web: tools to guide clients through the maze of fact and fiction. J Midwifery Womens Health 2002;47:264 – 8. California HealthCare Foundation. Ethics survey of consumer attitudes about health websites: new survey shows Internet users wary of health information privacy; California Health Care Foundation says industry wide guidelines necessary. Available at: http://www.chcf.org/press/ view.cfm?ItemID⫽12277. Accessed July 21, 2004. Goldman J, Hudson Z, Smith RM. Report on the privacy policies and practices of health websites. California HealthCare Foundation. Available at: http://www.chcf.org/topics/view.cfm?itemID⫽12497. Accessed July
544
Huang et al.
Quality of fertility clinic websites
21, 2004. 22. Sheppard S, Charnock D, Gann B. Helping patients access high quality health information. BMJ 1999;319:764 – 6. 23. Jones J. Development of a self-assessment method for patients to evaluate health information on the Internet. Proc AMIA Symp 1990: 540 – 4. 24. Jadad AR, Gagliardi A. Rating health information on the Internet: navigating to knowledge or to Babel? JAMA 1998;279:611– 4. 25. Hoffman-Goetz L, Clarke JN. Quality of breast cancer sites on the World Wide Web. Can J Pub Health 2000;91:281– 4. 26. Charnok D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Commun Health 1999;53:105–11. 27. Wyatt JC. Commentary: measuring quality and impact of the World Wide Web. Br Med J 1997;314:1879 – 81.
Vol. 83, No. 3, March 2005
Objective: To evaluate the overall quality of the Society for Assisted Reproductive Technology (SART)-affiliated fertility clinic websites, according to American Medical Association (AMA) Internet health information guidelines. Design: Cross-sectional evaluation. Setting: Two hundred sixty-six websites drawn from the SART directory and the World Wide Web. Intervention(s): Three objective scoring scales (ownership, content, and navigation) adapted from AMA guidelines. Main Outcome Measures: Seven objective criteria for ownership, 8 for content, and 11 for website navigation. Result(s): Two thirds of SART-affiliated fertility clinics have functional websites. Of the 236 sites evaluated, 58 belong to hospital centers. Overall, the scores for the three scoring scales were low. Compared with the websites of non-hospital clinics, those of hospital centers were more likely to include information about site ownership and affiliations (89.7% vs. 60.7%) and patient privacy (34.5% vs. 20.8%). Also, contents of hospital center websites were significantly easier to distinguish from advertisements (70.7% vs. 47.7%), and reference sources for specific contents were easier to identify (27.6% vs. 8.4%). Hospital center websites were more likely to indicate affiliations and financial disclosures of authors (25.9% vs. 10.7%), to feature a site map (25.9% vs. 12.3%), and to have a search function (31.0% vs. 5.6%). Conclusion(s): Websites of SART-affiliated clinics fail to meet most of the AMA health information guidelines. The quality of the hospital centers’ websites is better than that of private clinics. (Fertil Steril威 2005;83:538 – 44. ©2005 by American Society for Reproductive Medicine.) Key Words: Internet, World Wide Web, fertility, Society for Assisted Reproductive Technology (SART)
In recent years, the Internet has become more accessible to the general public, providing quick and convenient access to health information. Health-related websites represent one of the most common noncommercial web applications in the world (1). A vast amount of medical information is available through an estimated 20,000 to 100,000 health-related websites (2). In fact, the field of reproductive medicine is extensively represented on the World Wide Web. A recent search of the keyword “infertility” with the popular search-engine Google revealed 1,840,000 web pages (3). An estimated 52 million Americans browse the Internet to access health or medical information (4). Furthermore, studies done in Canada and The Netherlands showed that nearly half of infertile couples consult the Internet for fertilityrelated issues (5–7). We recently published a study looking at the reasons for Internet use and the source of online reproductive health information accessed by infertile couples at our institution (5). We found that 60% of women and 45% of men browse the Internet to obtain information on medical Received March 8, 2004; revised and accepted August 12, 2004. Reprint requests: Togas Tulandi, M.D., Department of Obstetrics and Gynecology, McGill University, 687 Pine Avenue West, Montreal, Que´bec H3A 1A1, Canada (FAX: 514-843-1448; E-mail: [email protected]).
538
conditions. In addition, 79% of men and 48% of women use the Internet to search for a second opinion from other fertility centers. More importantly, 62% of women and 41% of men stated that their decision to seek fertility treatments was influenced by health information on the Internet. As sources for online health information, academic institution or university websites (30.3%) and fertility clinic websites (28.1%) were the three most commonly cited sources (5). A popular reference is the Society for Assisted Reproductive Technology (SART) website, which provides a listing of 357 affiliated fertility clinics, including their contact information and website addresses (8). Unfortunately, the information provided on the Internet is often incomplete, misleading, or inaccurate (9). More specifically, the quality of online health information varies and might be biased by commercial interests. Furthermore, by accessing health information online, there is a risk of involuntarily revealing personal information or patterns of use and interests to third parties. To address these issues, the American Medical Association (AMA) published guidelines for medical and health information websites (10) to guide the development of web contents, to govern online advertising
Fertility and Sterility姞 Vol. 83, No. 3, March 2005 Copyright ©2005 American Society for Reproductive Medicine, Published by Elsevier Inc.
0015-0282/05/$30.00 doi:10.1016/j.fertnstert.2004.08.036
and sponsorships, and to protect the privacy and confidentiality of site visitors and patients. Currently, little is known about the quality of the fertility clinic websites in the United States. The purpose of our study was to evaluate the quality of the online health information from the SART-affiliated fertility clinics, using the AMA guidelines. MATERIALS AND METHODS A list of websites for SART-affiliated fertility clinics was obtained from the SART website (8). If a clinic’s website address was not listed in the SART directory, we used two search engines, Google (11) and Yahoo! (12), to search for the name of the fertility clinic and the medical director. Only clinics with websites were reviewed. Two coauthors (J.Y.J.H. and F.D.) evaluated each website. The fertility clinic websites were classified according to the practice settings (hospital-sponsored centers or independently operated clinics) and the site affiliations (universities, other health information websites, and pharmaceutical companies). A website was considered to be hospital sponsored if the affiliation was clearly stated in the web pages. Evaluation Criteria The quality of online health information was assessed according to three objective scoring scales adapted from the AMA online health information guidelines (ownership and affiliation, editorial content, and navigation of web contents) (10). We developed seven objective criteria to assess information regarding website ownership and affiliations, with a score of 1 given for each of the criteria met, for a maximum score of 7: 1. Website ownership, including affiliations, strategic alliances, and significant investors, should be clearly indicated on the home screen or via a directly accessible link on the home screen. 2. Copyright ownership of specific content should be clearly indicated. 3. Information about restrictions on access to content, required registration, and password protection, if applicable, should be provided and easy to find. 4. Information about payment subscriptions, document delivery, and viewing should be provided and easy to find. 5. Information about privacy should be provided and easy to find. 6. Funding or other sponsorship for any specific content should be clearly indicated. 7. Content should be easily distinguished from advertising. Editorial content was assessed according to eight objective criteria, with a score of 1 given for each of the criteria met, for a maximum score of 8: Fertility and Sterility姞
1. Content should be reviewed for quality (including originality, accuracy, and reliability) by either peer review or editorial boards. 2. The language complexity of the content should be appropriate for the site’s audience. 3. A description of the editorial process and method of content review should be posted on the site. 4. All staff members and other individuals (i.e., editorial board) responsible for content quality should be posted on site. 5. The dates on which content is posted, revised, and updated should be clearly indicated. 6. Sources for specific content should be clearly identified (i.e., author byline or names of individual, organization, department, institution, agency, or commercial provider/ producer). 7. Affiliations and relevant financial disclosures for authors and content producers should be clearly indicated. 8. Reference material used to develop content should be cited in a manner appropriate for the site’s audience. Navigation of web contents was evaluated according to 11 specific criteria, with a score of 1 given for each of the criteria met, for a maximum score of 11: 1. The website should provide information about the platform(s) and browser (that permit optimal viewing in a location that is easy to find). 2. The intrasite content links should be functional. 3. The external links should be functional. 4. The website allows viewers to return to previously browsed websites. 5. The website does not redirect viewer to unintended websites. 6. Information on portable document format (PDF) files and software download instruction should be provided and easy to find. 7. The website features include a site map. 8. The website features include a “frequently asked questions” (FAQ) page. 9. The website features include a feedback mechanism. 10. The website features include customer service information. 11. Each distinct site should provide a search engine with instructions specifying how to use the search function. The two evaluators tested all the intrasite and external hyperlinks of each website. All links had to be functional to be awarded 1 point. Partial credits were not awarded. Statistical Analysis For comparison between hospital-sponsored centers and private clinics, and between university-affiliated clinics and non– university-affiliated clinics, we used the 2 test for categoric variables and the Mann-Whitney test for continuous variables. Results are expressed as mean and 95% confidence interval; the differences were considered statistically significant if P⬍.05. 539
TABLE 1 Website ownership information.
All websites Hospitals (n ⴝ 236) (n ⴝ 58) Website ownership, affiliations, strategic alliances, and significant investors Copyright ownership Restrictions of access to content, required registration, and password protection Payment subscription, document delivery, and viewing Privacy protection Funding or sponsorship information Content is easily distinguished from advertising
160 (67.8)
Clinics (n ⴝ 178)
52a (89.6) 108a (60.7)
Universityaffiliated centers (n ⴝ 52)
Non–universityaffiliated centers (n ⴝ 184)
46a (88.5)
114a (61.9)
91 (38.5) 17 (7.2)
19 (32.7) 5 (8.6)
73 (41.0) 12 (6.7)
18 (34.6) 3 (5.7)
74 (40.2) 14 (7.6)
44 (18.6)
7 (12.1)
37 (20.8)
6 (11.5)
38 (20.6)
56 (23.7) 11 (4.7)
20b (34.5) 5 (8.6)
37b (20.8) 6 (3.4)
15 (28.8) 5 (9.6)
42 (22.8) 6 (3.3)
126 (53.4)
41b (70.7)
85b (47.7)
37b (71.1)
89b (48.4)
Note: Data are presented as n (%). a P⬍.001. b P⬍.05. Huang. Quality of fertility clinic websites. Fertil Steril 2005.
RESULTS Two thirds (66.1%) of SART-affiliated fertility clinics in the United States had functional websites. Among the 236 websites evaluated, 58 centers were sponsored by hospitals and 178 by independently operated fertility clinics. Twenty-two percent of the websites were affiliated with academic institutions, and 20.3% of them were affiliated with other health information websites.
reliability by either peer review or editorial boards. Only 2.5% of websites provided a description of the editorial process. Just 13.1% of websites identified the source of specific contents, and 16.5% of websites cited reference materials in a manner appropriate for the site’s audience. As for the timeliness of the web contents, only 11.4% of websites posted the dates of the content publications and revisions.
Overall, the scores for the three scoring scales were low. The mean score for the 7-point assessment of site ownership was 2.1. The mean score for the 8-point assessment of quality of website content was 2.1. The mean score for the 11-point evaluation of website navigation functions was 6.0.
Regarding the navigational functionality of the websites, most websites featured functional intrasite and external links, allowed viewers to return to a previously browsed web page, and did not redirect viewers to unintended websites (Table 3). Customer feedback mechanisms existed in 93.2% of websites. However, most websites lacked functions such as a site map, a FAQ page, a search engine, and download instructions for PDF files.
Regarding website ownership information, 67.8% of websites clearly indicated information about ownership, including affiliations, strategic alliances, and significant investors (Table 1). However, only 38.5% of websites indicated copyright information of specific web contents. More importantly, only 23.7% of websites included information about patient privacy policies, and a mere 4.7% of websites revealed funding or other sponsorship for any specific content. Regarding the quality of the website content, 50% of websites listed all staff members and individuals responsible for content quality (Table 2). However, only 3.4% of the websites evaluated attested that their web contents were reviewed for quality, including originality, accuracy, and 540
Huang et al.
Quality of fertility clinic websites
Comparisons Between Hospital-Sponsored Centers and Private Clinics Compared with private clinics, hospital-sponsored center websites were more likely to include information about ownership and affiliation (89.7% vs. 60.7%, P⬍.001) and patient privacy (34.5% vs. 20.8%, P⬍.05). Contents of the hospital center websites were significantly easier to distinguish from advertisements (70.7% vs. 47.8%, P⬍.05). Reference sources for specific contents were significantly easier to identify in hospital center websites (27.6% vs. 8.4%, Vol. 83, No. 3, March 2005
TABLE 2 Website content.
Content is reviewed by editorial boards Appropriate language complexity Editorial process description Individuals responsible for content are posted on site The content is revised and updated regularly Source of content is identified Affiliations and relevant financial disclosures for authors Reference material is cited
All websites (n ⴝ 236)
Hospitals (n ⴝ 58)
Clinics (n ⴝ 178)
Universityaffiliated centers (n ⴝ 52)
Non–universityaffiliated centers (n ⴝ 184)
8 (3.4)
1 (1.7)
7 (3.9)
2 (3.8)
6 (3.3)
233 (98.7)
58 (100)
176 (98.8)
52 (100)
182 (98.9)
6 (2.5) 118 (50.0)
0 (0) 25 (43.1)
6 (3.4) 94 (52.8)
0 (0) 25 (48.1)
6 (3.3) 94 (51.1)
27 (11.4)
10 (17.2)
17 (9.6)
9 (17.3)
18 (9.8)
31 (13.1) 34 (14.4)
16a (27.6) 15b (25.9)
15a (8.4) 19b (10.7)
15a (28.8) 17a (32.7)
16a (8.7) 17a (9.2)
39 (16.5)
13 (22.4)
26 (14.6)
13 (25)
26 (14.1)
Note: Data are presented as n (%). a P⬍.001. b P⬍.05. Huang. Quality of fertility clinic websites. Fertil Steril 2005.
P⬍.001). Also, hospital center websites were more likely to indicate affiliations and financial disclosures of authors (25.9% vs. 10.7%, P⬍.05), to feature a site map (25.9% vs. 12.4%, P⬍.05), and to have a search function (31.0% vs. 5.6%, P⬍.001). Comparison Between University-Affiliated Clinics and Non–University-Affiliated Clinics University-affiliated centers were more likely to include ownership and affiliation information (88.5% vs. 62.0%, P⬍.001) than non-university centers. Web contents were significantly easier to distinguish from advertisements (71.2% vs. 48.4%, P⬍.05), and reference sources were significantly easier to identify (28.8% vs. 8.7%, P⬍.001). Also, university-affiliated clinic websites were more likely to indicate relevant financial disclosures of authors (32.7% vs. 9.2%, P⬍.001) and to include a search function (26.9% vs. 7.6%, P⬍.001). DISCUSSION This is the first study to evaluate the quality of the online health information from SART-affiliated fertility clinics, according to objective criteria based on the AMA guidelines. We found that two thirds of the SART-affiliated fertility clinics have functional websites. Irrespective of the practice setting or university affiliation, the overall quality of the fertility clinic websites is poor, failing to meet most of the Fertility and Sterility姞
AMA Internet health information guidelines. Most websites failed to provide information about copyright or sponsorship of any specific content, payment subscriptions or methods of document delivery and viewing, and the protection of viewer privacy. Regarding the quality of website content, most sites lacked descriptions of the editorial process, the source of specific contents, the proper citations of reference sources, the dates on which the contents were posted, revised, and updated, and relevant financial disclosures of authors and content producers. Although most websites featured functional links, the majority of websites lacked features such as a site map, a FAQ page, and search functions. Moreover, most websites did not provide clear information about the web browser recommended for optimal viewing or instructions for downloading PDF document files. Owing to the lack of a peer-review process, such as the one used for medical journals, the validity of the information on the Internet is sometimes questionable. Some of these health-related sites are created and maintained by academic institutions, some by healthcare groups (2), and others by commercial companies with the intent to advertise a product. A large part of the material is helpful, but one often finds incorrect or incomplete information (13). Furthermore, the language complexity and the clarity of the information are not standardized and might not be suitable for individuals without proper medical backgrounds (14). 541
TABLE 3 Website navigation. University-affiliated All websites Hospitals Clinics centers (n ⴝ 236) (n ⴝ 58) (n ⴝ 178) (n ⴝ 52) The site provides information about the browser Functional intrasite links Functional external links Viewers allowed to return to previous websites The website does not redirect viewer to unintended websites Portable document format file (PDF) download instructions are provided and easy to find Features a site map Features “frequently asked questions” page Features feedback mechanism Features customer service information Provides a search engine
5 (2.1)
0 (0)
5 (2.7)
0 (0)
Non–universityaffiliated centers (n ⴝ 184) 5 (2.7)
227 (96.2) 229 (97.0) 206 (87.3)
57 (98.3) 171 (96.1) 57 (98.3) 173 (97.2) 51 (87.9) 156 (87.6)
51 (98.1) 51 (98.1) 45 (86.5)
177 (96.2) 179 (97.3) 162 (88.0)
227 (96.2)
56 (96.5) 172 (96.6)
50 (96.1)
178 (96.7)
20 (11.2)
7 (13.5)
20 (10.9)
36 (15.2) 77 (32.6)
15b (25.9) 22b (12.3) 17 (29.3) 61 (34.3)
12 (23.1) 17 (32.7)
25 (13.6) 61 (33.1)
220 (93.2)
55 (94.8) 166 (93.2)
50 (96.1)
171 (92.9)
119 (50.4)
32 (55.2)
27 (51.9)
92 (50)
14a (26.9)
14a (7.6)
27 (11.4)
28 (11.9)
7 (12.1)
87 (48.9)
18a (31.0) 10a (5.6)
Note: Data are presented as n (%). a P⬍.001. b P⬍.05. Huang. Quality of fertility clinic websites. Fertil Steril 2005.
In 1999, the American Society for Reproductive Medicine published guidelines for advertising by assisted reproductive technology (ART) programs. However, the guidelines are limited to the advertisement of successful pregnancy rates from ART (15). Several other health organizations have developed generic rating tools to objectively evaluate online health information, including the Health On the Net (HON) Foundation and the British Healthcare Internet Association (16, 17). These guidelines are similar to the Journal of the American Medical Association’s minimal core standards for responsible printed documents, including authorship of the content, attribution to the sources of the content, disclosure of funding and competing interests, and timeliness of the information presented (9). Unfortunately, we discovered that even websites that displayed the HON insignia did not always comply with all AMA guidelines. This finding is even more disturbing because the HON insignia implies that the website conforms to its quality assurance criteria. This is an alarming trend, which has been noted in other healthcare literature. For example, a study evaluating the quality of 542
Huang et al.
Quality of fertility clinic websites
online asthma information also reported websites with the HON insignia that did not conform to all the HON guidelines (18). We compared websites sponsored by hospital centers and by independently operated clinics to evaluate one of our research hypotheses: the quality of hospital center websites is superior to that of private clinics. Hospitals often have more elaborate websites and might have stricter guidelines for publication of health information. Similarly, we believe that websites affiliated with academic institutions are more extensive in content and frequently include links to health research and educational websites. Findings from this study are similar to those from other studies evaluating the quality of health information on the Internet. Okamura et al. (19) evaluated 197 fertility-related websites, using the AMA’s four minimal core standards for responsible printed documents. The investigators found that only 2% of websites met all four recommended standards, and up to 50% of websites failed to report any of the four Vol. 83, No. 3, March 2005
standards. They concluded that commercial websites were more likely than educational websites to fail to meet minimum standards. In contrast, we did not find any difference between the quality of academic institution–affiliated clinic websites and private clinic websites. Nonetheless, the quality of Internet resources available to infertile couples is at best variable. A survey of online health users cited the protection of personal privacy as the most important priority (20). Current technology allows the host site or third parties to collect personal information over the Internet and to track health information–seeking behaviors, which could breach personal privacy and reveal the user’s health data. In the present study, we found that only 23.7% of websites feature patient privacy protection statements. This is an alarming finding because the percentage might be actually an overestimation. It has been shown in previous studies that although many health information websites state privacy policies, most of the websites do not follow them (21). There are some limitations to our study. It was based on American websites listed on the SART website (8), and not on websites listed according to search engines. It is possible that the 236 websites surveyed were not among the most popular websites visited by fertility couples. Moreover, some clinics might have websites that were not accessible with our search methods. We did not actually contact the clinics and ask whether they have functional websites. However, we believe that our method of using websites recognized by SART is reasonable because there are few reliable and complete lists of fertility clinic websites. Furthermore, SART includes more than 370 practicing members, representing more than 95% of the ART clinics in the nation. Second, the World Wide Web is a constantly evolving technology, and the information available on the Internet is constantly being altered. In any event, we believe that our study reflects the most current state of the technology and practice on the World Wide Web. Although numerous health organizations have published instruments and guidelines to evaluate online health information, the reliability and validity of these tools have been questioned (16, 22–24). Our purpose was to evaluate information from SART-affiliated websites according to a set of objective scoring criteria. We chose the AMA Guidelines for Medical and Health Information Sites over other rating tools because it has been adopted by the Health Summit Working Group and has been used in numerous studies (25). Last, the interrater reliability was not assessed. Nevertheless, because the websites were evaluated according to a set of objective criteria, we believe that interrater reliability should not be a significant factor in this study. The widespread use of the Internet by patients poses a modern challenge to obstetricians and gynecologists. Patients are taking a more active role in deciding about their health care (26) and have realized that information from doctors is often inadequate to satisfy their desire for knowlFertility and Sterility姞
edge regarding health (5, 27). Furthermore, 21% of Internet users prefer being followed by physicians with their own websites (9). Physicians must anticipate that patients might enter their practices with information concerning their medical conditions or treatments obtained from the Internet. A standard patient visit should therefore include questions regarding what the patients are learning from their Internet searches for health information and educating patients on how to better evaluate quality of information on the Internet. Physicians must assume a new role of health information provider, maintaining their own health websites with accurate and current information or directing their patients to reliable and up-to-date Internet health websites. More importantly, professional organizations governing the practice of ART should develop more specific guidelines on online health information to ensure that patients have access to the most reliable, current, and high-quality health information on reproductive medicine. REFERENCES 1. Lawrence S, Giles CL. Accessibility of information on the Web. Nature 1999;400:107–9. 2. Eysenbach G, Sa ER, Diepgen TL. Shopping around the Internet today and tomorrow: towards the millennium of cybermedicine. BMJ 1999; 319:1294 – 8. 3. Google. Search results for term “infertility.” Available at: www.google. com. Accessed July 21, 2004. 4. Fox S, Rainee L, Horrigan J, Lenhart A, Spooner T, Burke M, et al. The online healthcare revolution: how the Web helps Americans take better care of themselves. Washington, DC: Pew Internet and American Life Project. Available at: http://www.pewinternet.org. Accessed July 21, 2004. 5. Huang JYJ, Al-Fozan H, Tan SL, Tulandi T. Internet use by patients seeking fertility treatment. Int J Gynecol Obstet 2003;83:75– 6. 6. Haagen EC, Tuil W, de Bruijn RP, Braat DDM, Kremer JAM. Current Internet use and preferences of IVF and ICSI patients. Hum Rep 2003;18:2073– 8. 7. Weissman A, Gotlieb L, Ward S, Greenblatt E, Casper R. Use of Internet by infertile couples. Fertil Steril 2000;73:1180 –2. 8. Society for Assisted Reproductive Technology. Available at: www. sart.org. Accessed April 1–15, 2003. 9. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling and assuring the quality of medical information on the Internet: caveant lector et viewor—let the reader and viewer beware. JAMA 1997;277: 1244 –5. 10. Winker MA, Flanagin A, Chi-Lum B, White J, Andrews K, Kennett RL, et al. Guidelines for medical and health information sites on the Internet. JAMA 2000;283:1600 – 6. 11. Google. Available at: www.google.com. Accessed April 1–15, 2003. 12. Yahoo! Available at: http://www.yahoo.com. Accessed April 1–15, 2003. 13. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the World Wide Web: systematic survey of advice on managing fever in children at home. BMJ 1997;314: 1875–9. 14. Graber MA, Roller CM, Kaeble B. Readability levels of patient education material on the World Wide Web. J Fam Pract 1999;48:58 – 61. 15. American Society for Reproductive Medicine. A practice committee report. Guidelines for advertising by ART programs 1999. Available at: http://www.asrm.org/membersonly/practice/ArtAdvertising.pdf. Accessed July 21, 2004. 16. Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related websites: review. BMJ 1999;318:647–9. 17. Health on the Net Foundation. Health on the Net code of conduct. Avail-
543
18. 19.
20.
21.
able at: http://www.hon.ch/honcode/honcode_check.html. Accessed July 21, 2004. Croft DR, Peterson MW. An evaluation of the quality and contents of asthma education on the World Wide Web. Chest 2002;121:1301–7. Okamura K, Bernstein J, Fidler AT. Assessing the quality of fertility resources on the World Wide Web: tools to guide clients through the maze of fact and fiction. J Midwifery Womens Health 2002;47:264 – 8. California HealthCare Foundation. Ethics survey of consumer attitudes about health websites: new survey shows Internet users wary of health information privacy; California Health Care Foundation says industry wide guidelines necessary. Available at: http://www.chcf.org/press/ view.cfm?ItemID⫽12277. Accessed July 21, 2004. Goldman J, Hudson Z, Smith RM. Report on the privacy policies and practices of health websites. California HealthCare Foundation. Available at: http://www.chcf.org/topics/view.cfm?itemID⫽12497. Accessed July
544
Huang et al.
Quality of fertility clinic websites
21, 2004. 22. Sheppard S, Charnock D, Gann B. Helping patients access high quality health information. BMJ 1999;319:764 – 6. 23. Jones J. Development of a self-assessment method for patients to evaluate health information on the Internet. Proc AMIA Symp 1990: 540 – 4. 24. Jadad AR, Gagliardi A. Rating health information on the Internet: navigating to knowledge or to Babel? JAMA 1998;279:611– 4. 25. Hoffman-Goetz L, Clarke JN. Quality of breast cancer sites on the World Wide Web. Can J Pub Health 2000;91:281– 4. 26. Charnok D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Commun Health 1999;53:105–11. 27. Wyatt JC. Commentary: measuring quality and impact of the World Wide Web. Br Med J 1997;314:1879 – 81.
Vol. 83, No. 3, March 2005