- Email: [email protected]
An Evaluation of the Quality and Contents of Asthma Education on the World Wide Web
special report An Evaluation of the Quality and Contents of Asthma Education on the World Wide Web* Donita R. Croft, MD, MS; and Michael W. Peterson, MD, FCCP
Study objectives: To measure the accessibility and quality of currently available asthma education World Wide Web sites using the following criteria: accessibility by readability, language, and download time; information quality based on inclusion of core educational concepts and compliance with Health On the Net (HON) principles; and utilization of innovative technology. Design: Objective evaluation of 145 Web sites. Measurements and results: Four search engines or directories (Yahoo, HON, Alta Vista, and Healthfinder) were searched for “asthma, patient information.” A maximum of 50 Web sites from each search engine or directory was evaluated. Only 90 of the 145 Web sites actually contained asthma educational material. The mean (ⴞ SD) time necessary to open each Web site on a 28.800-bits-per-second modem was 33.6 ( ⴞ 36.6) s. The mean number of graphics on the Web sites was 24.6 ( ⴞ 30.2) files per page. The educational material required a mean reading level beyond the 10th grade. Only nine Web sites contained multilingual asthma education material. The mean number of HON principles with which the Web sites conformed was 6.3 ( ⴞ 1.0) of 8 principles; 14 Web sites conformed to all the HON criteria. The average Web site contained 4.9 (ⴞ 2.5) of 8 core asthma educational concepts, and only 20 Web sites contained all 8 educational concepts. Very few Web sites utilized innovative educational technology. Conclusions: While patient asthma education Web sites are common, asthma educational material contains many accessibility barriers, is highly variable in quality and content, and takes little innovative use of technology. Patient educational material currently available on the World Wide Web fails to meet the information needs of patients. (CHEST 2002; 121:1301–1307) Key words: asthma; information distribution; Internet; language; learning; lung disease; patient education; reading; technology; World Wide Web Abbreviations: AMA ⫽ American Medical Association; bps ⫽ bits per second; HON ⫽ Health On the Net; HONcode ⫽ Health On the Net Code of Conduct; NAEP ⫽ National Asthma Education and Prevention Program
of the World Wide Web compoT henentintroduction of the Internet has made information available to people on an unprecedented scale. It is estimated that 360 million people worldwide cur*From the Department of Internal Medicine, University of Iowa College of Medicine, Iowa City, IA. Dr. Croft is currently an Epidemic Intelligence Service Officer with the Centers for Disease Control and Prevention, Atlanta, GA. This study did not receive financial support from any company or organization. The opinions contained herein are the views of the authors and are not meant to represent the views of any agency or institution. Manuscript received April 9, 2001; revision accepted October 30, 2001. Correspondence to: Donita R. Croft, MD, MS, PO Box 2659, Madison, WI 53701; e-mail: [email protected] www.chestjournal.org
rently have access to the World Wide Web.1 The World Wide Web contains about 800 million publicly available pages, and approximately 3% are health-related Web sites.2 The World Wide Web is For editorial comment see page 1015 potentially a very powerful vehicle for providing patient education. It has several advantages as a health-related educational tool for patients. First, it is relatively inexpensive to disseminate information to millions of people, even across national borders. Second, the World Wide Web can be used to link people with similar conditions, which is especially CHEST / 121 / 4 / APRIL, 2002
1301
important for people who are homebound, live in geographically remote areas, or have rare conditions. Third, the design and structure of the Internet make it difficult to disrupt transmission of information. Finally, computer software allows for significant interactivity with the user and provides the opportunity for users to be actively engaged in learning. Modern adult learning theory suggests that active participation by the learner significantly enhances the learning.3 However, the World Wide Web also presents unique new challenges as an effective patient educational vehicle. The very structure of the Internet leads to a very decentralized control. It is therefore difficult to enforce quality-control measures to ensure that the information provided is accurate. Peer review has been traditionally used for quality control of medical publications; however, this is much more problematic on the World Wide Web. Because of the ease with which any individual can launch a Web site, inaccurate or even deceptive material can be transmitted to vast numbers of people before it is identified as such. It is difficult for patients and professionals alike to judge the quality of information. Several organizations, institutions, and journals have developed generic rating tools in an attempt to objectively evaluate medical information on the World Wide Web. The most commonly used principles in these rating tools are disclosure of the authors and sponsors, information currency, accessibility, and ease of use of the information. The American Medical Association (AMA), British Healthcare Internet Association, and Health On the Net (HON) Foundation are examples of groups that have developed such tools.4 In addition to quality control, accessibility is a potential problem on Web-based educational sites. Accessibility can be viewed both in technical terms and in communication terms. In technical terms, this is often referred to as the “last mile” problem because most World Wide Web users currently access the system using modems with limited communication capacity. Since users have a limited capacity for waiting for information, poorly designed pages with extensive graphics present a barrier for delivering information effectively. Another limitation is communication barriers. The majority of information on the World Wide Web is written in English only, which limits patient access given the worldwide reach of the media.5 Even for English-speaking users, the readability level of the information can be a barrier. In evaluating 50 healthrelated Web sites, Graber et al6 found the average reading level at about a 10th-grade level despite the fact that the level of reading comprehension is considerably lower in most patient populations. 1302
The World Wide Web thus presents both opportunities and challenges for patient education. Patient self-management, however, is an important component of chronic disease management. It is widely believed that a patient who is educated about his or her health-related problem will better adhere to treatment plans and have improved health outcomes. Patients also want information about their own health and management. It has been estimated that between 30% and 50% of Internet users search for health-related information on the World Wide Web on a regular basis.7 In pulmonary medicine, we deal with several chronic diseases in our patient population. Asthma is probably the best-studied pulmonary disease model in which patient self-management is incorporated. Several studies8 –11 have reported that asthma patient education programs improve compliance with medications, decrease emergency department visits and hospitalizations, and improve quality of life. Formal asthma education programs have also been shown to be cost-effective.12 Recognizing this, patient education is an integral component in the National Asthma Education and Prevention Program (NAEP) Expert Panel Report 2.13 However, formal asthma education programs are costly, time-consuming, and require significant expertise. Consequently, even many training programs in pulmonary medicine do not incorporate formal patient education programs into their asthma care.14 The World Wide Web presents itself as a potential tool for solving this problem; indeed, many asthma patient education sites are now available on the World Wide Web. Given the challenges presented by the World Wide Web as a quality education tool, however, we know very little about the nature of the patient educational materials available on asthma management. We undertook this study to measure the quality of currently available Internet sites using accessibility, information quality, and innovative design as our criteria. Based on previous studies of both patient and professional information offered on the Internet, we postulated that asthma patient information would be highly variable in each of these domains. Materials and Methods Four search engines or directories (Yahoo, Alta Vista, Healthfinder and HON) were chosen as the sources of the Internet sites. These were chosen to provide a diverse variety of healthrelated and non-health-related search engines and directories. Using the key words “asthma” and “patient information” or “asthma, patient information,” a search for Web sites was undertaken in each search engine or directory. The total number of Web sites produced by the searches was recorded. The first 50 Web sites from each search engine or directory were consecuSpecial Report
tively evaluated. If ⬍ 50 Web sites were produced by the search, all sites were evaluated. Web sites evaluated in one search were excluded from subsequent search result evaluations. Sites that did not include actual asthma patient information were also excluded. The accessibility of information on the Web sites was assessed by several different means. As a measure of the time necessary to access each Web site, FrontPage 2000 (Microsoft; Redmond, WA) was used to determine the amount of time required to load the homepage of the Web site using a 28.800-bits-per-second (bps) modem. In addition, because the number of graphics in a Web site is directly related to the amount of time needed to open the site, the graphics in the Web sites were quantified. This was accomplished by opening each site in its hypertext markup language source and counting the number of .gif and .jpg files (the most common file extensions for graphical files) included on the home page. The readability of each site was determined by copying the asthma education material from the site, pasting it into a Microsoft Word 97 (Microsoft) document, and obtaining the Flesch score and Flesch-Kincaid readability grade level. The Flesch score ranges from 0 to 100, from the most difficult to the easiest to read, respectively.6 The Flesch-Kincaid readability grade level determines the difficulty of the material by measuring the length of the words and sentences and converting the results into a reading level from third to 12th grade.15 The FleschKincaid readability grade level is unable to discriminate material less complex than the third grade or more complicated than the 12th grade. As a final means of determining the accessibility of information on the Web sites, the language used in each site was reviewed; if at least some of the educational materials were in more than one language, the Web site was identified as multilingual. Eight core educational concepts were identified by comparing the asthma education material of the NAEP,13 the American Lung Association,16 and the AMA.17 The eight asthma educational concepts that the three groups had in common were used as core educational concepts. These concepts are listed in Table 1. The educational material on the searched Web sites was evaluated, and inclusion of any of the eight concepts was quantified. The use of innovative technology on the Web sites was evaluated by reviewing the sites and recording the use of multimedia capability, interactive activities, personalizable material, support groups, and presence of synchronous or asynchronous communication to answer specific questions posed by the user. The use of multimedia capability was defined as the application of video or audio media for the purpose of teaching about asthma. The Web site was recorded as being interactive if it contained activities in which the responses varied, depending on the information provided by the user. These activities included tests or quizzes in which responses were tailored to the provided answers. Information that changed in response to data
Table 1—Core Asthma Educational Concepts Used to Evaluate Content of Asthma Patient Education Web Sites 1. 2. 3. 4. 5. 6. 7. 8.
Basic asthma pathophysiology Triggers of asthma attacks Avoidance of asthma triggers Mechanism of action of medications for asthma Importance of use of anti-inflammatory medications Importance of availability of rescue medications Self-assessment of respiratory status Necessity of an action plan dependent on respiratory status
www.chestjournal.org
specific to the reader’s condition or symptoms was considered personalizable material. An example of personalizable material is recommendations on the use of metered-dose inhalers based on peak flow readings. Support groups were defined as a mechanism by which the reader could be in contact with other users of the Web site. The Web sites were also reviewed for the capacity to answer asthma-related questions asked by the reader via e-mail or forms. The disclosure of the credentials of the person answering the questions was noted. Also, the sponsorship of the site was recorded in the following categories: commercial, government, university, organization, individual, or unknown. This information was obtained from sponsorship information, Web site content or, if not obvious in the site, from the address of the Web site. For example, if the address ended in “.com,” it was considered a commercial site. The quality of content was evaluated using the Site-Checker of the HON Code of Conduct (HONcode).18 This quality rating tool was used because the HONcode is comprehensive and the Site-Checker is easily navigated. The HONcode requires the identity and credentials of the author, adherence to confidentiality, timeliness of modifications, and disclosure of the funding source of the Web sites. The HONcode principles are listed in Table 2. The presence or absence of the HON insignia was recorded. The HON insignia is available for placement on Web
Table 2—HONcode Principles* 1. Any medical/health advice provided and hosted on this site will only be given by medically or health-trained and qualified professionals unless a clear statement is made that a piece of advice offered is from a nonmedically/health qualified individual/organization. 2. The information provided on this site is designed to support, not replace, the relationship that exists between a patient/site visitor and his/her existing physician. 3. Confidentiality of data relating to individual patients and visitors to a medical/health Web site, including their identity, is respected by this Web site. The Web site owners undertake to honor or exceed the legal requirements of medical/health information privacy that apply in the country and state where the Web site and mirror sites are located. 4. Where appropriate, information contained on this site will be supported by clear references to source data and, where possible, have specific HTML links to that data. The date when a clinical page was last modified will be clearly displayed (eg, at the bottom of the page). 5. Any claims relating to the benefits/performance of a specific treatment, commercial product or service will be supported by appropriate, balanced evidence in the manner outlined above in principle 4. 6. The designers of this Web site will seek to provide information in the clearest possible manner and provide contact addresses for visitors who seek further information or support. The Webmaster will display his/her e-mail address clearly throughout the Web site. 7. Support for this Web site will be clearly identified, including the identities of commercial and noncommercial organizations that have contributed funding, services or material for the site. 8. If advertising is a source of funding it will be clearly stated. A brief description of the advertising policy adopted by the Web site owners will be displayed on the site. Advertising and other promotional material will be presented to viewers in a manner and context that facilitates differentiation between it and the original material created by the institution operating the site. *HTML ⫽ hypertext markup language. CHEST / 121 / 4 / APRIL, 2002
1303
sites based on an honor system and is meant to indicate that the site adheres to all of the HON principles. In addition, obviously inaccurate information on the Web sites was noted. The data were recorded and analyzed using Microsoft Access 97 (Microsoft).
Results Between November 1999 and February 2000, we identified and evaluated 145 Web sites. The search of HON revealed 950 Web sites; Yahoo revealed 30 Web sites; Alta Vista revealed 245,630 Web sites; and Healthfinder revealed 87 Web sites. From these groups, 50 Web sites were evaluated from HON, 30 from Yahoo, 50 from Alta Vista, and 15 from Healthfinder. We evaluated ⬍ 50 sites from Healthfinder because they represented duplicate Web sites that we evaluated using other search engines. Table 3 displays characteristics of the Web sites. Five Web sites were inaccessible either because they were technically unavailable at the time of the search (n ⫽ 3) or the sites were not written in English (n ⫽ 2). Of the 140 accessible Web sites, only 90 sites (64.3%) actually contained asthma education material. The Web sites that did not contain asthma education material contained information about product sales, research recruitment, obtaining an appointment in a doctor’s practice, or conference schedules, or the sites were simply portals to other Web sites containing asthma education material. Accessibility The mean time required to open the Web sites on a 28.800-bps modem, as determined by FrontPage 2000, was 33.6 (⫾ 36.6) s, and the median was 25 s (range, 0 to 292 s). This result was based on 112 Web sites because 33 of the Web sites would not allow
Table 3—Web Site Characteristics Characteristics Accessibility Accessible during search Inaccessible during search Could not read language of the Web site Asthma education material Contained Not contained Sponsorship Commercial Organization University Government Individual Unknown
1304
FrontPage 2000 to open them in an editor. The mean number of graphics located on the Web sites was 24.6 ( ⫾ 30.2) files per page, and the median was 16 files per page (range, 0 to 206 files per page). The educational material had a mean Flesch score of 46.3 ( ⫾ 13.5) and a median of 47.6 (range, 8.9 to 75.7). The mean Flesch-Kincaid readability level of these Web sites was 10.3 (or 10th grade, third month) [⫾ 1.7], and the median was 10.5 (range, 5.2 to 12.0.) When stratified by sponsorship, the mean FleschKincaid readability level ranged from 10.2 (company and organization) to 10.9 (individual). Twenty-seven of the Web sites scored at the maximum readability, 12.0. Only nine Web sites contained multilingual asthma education material. These Web sites were written in Spanish and English (n ⫽ 2); French and English (n ⫽ 2); Italian, French, and English (n ⫽ 1); Spanish, French and English (n ⫽ 1); German and English (n ⫽ 1); Korean and English (n ⫽ 1); and Finnish and English (n ⫽ 1). Of the Web sites that were written in a single non-English language, one was in Spanish only and one in Turkish only. Quality Control We first quantified generic quality by determining the adherence of the Web site to HONcode principles. The mean number of HON principles followed in the Web sites with asthma education material was 6.3 (⫾ 1.0) principles, and the median was 6 principles (range, 4 to 8 principles). Table 4 shows the number of Web sites conforming to each of the HON principles. Only 14 of the Web sites conformed to all eight of the HON criteria. The HON insignia was present in 16 of the Web sites containing asthma education material. Of these 16 Web sites, only 6 sites conformed to all of the HON principles. One Web site sponsored by a university erroneously stated that Serevent (GlaxoSmithKline; Research Triangle Park, NC) is a quick-acting medi-
No. of Web Sites 140 3 2 90 50 67 36 22 10 8 2
Table 4 —Web Sites Conforming to the HON Principles Principle No.
No. of Web Sites (%)
1 2 3 4 5 6 7 8 All 8
79 (87.8) 90 (100.0) 90 (100.0) 24 (26.7) 89 (98.9) 73 (81.1) 34 (37.8) 87 (96.7) 14 (15.6)
Special Report
cine. The investigators e-mailed the two contacts listed on the Web site about this incorrect information. One e-mail was returned undeliverable, and no response was received from the second contact. The misinformation remained on the Web site several months later. Five of the Web sites were based entirely on practices of complementary medicine, whereas many sites mentioned alternative medicine practices. We next evaluated the sites based on whether they provided the eight core educational components. Of the Web sites containing asthma education material, the mean number of core educational concepts was 4.9 (⫾ 2.5) out of 8, and the median was 5 concepts (range, 1 to 8 concepts). The number of concepts included in the Web sites, when stratified by sponsorship of the Web site, is displayed in Table 5. The government-sponsored sites contained the most core concepts, and the individual-sponsored sites contained the least core educational concepts. Twenty Web sites contained all eight core educational concepts. These were sponsored by organizations (n ⫽ 8), commercial groups (n ⫽ 7), government (n ⫽ 4), and a university (n ⫽ 1). Table 6 lists the core educational concepts and their frequency of inclusion in the Web sites. The most frequently included educational concept was triggers of asthma attacks (90.1%), and the least commonly included concept was the importance of the availability of “rescue” medications (37.4%).
Table 6 —Frequency of Inclusion of Core Educational Concepts Educational Concepts Basic asthma pathophysiology Triggers of asthma attacks Avoidance of asthma triggers Mechanism of action of medications for asthma Importance of use of anti-inflammatory medications Importance of availability of rescue medications Self-assessment of respiratory status Necessity of an action plan dependent on respiratory status
No. of Websites (%) 76 (84.4) 82 (91.1) 65 (72.2) 54 (60.0) 50 (55.6) 34 (37.8) 42 (46.7) 39 (43.3)
were commercially sponsored. In 20 of the Web sites, the reader had the capacity to e-mail an asthma-related question. These Web sites reported that the questions were answered by a physician (n ⫽ 11), a physician or a pharmacist (n ⫽ 1), a registered nurse (n ⫽ 1), a nurse practitioner (n ⫽ 1), a respiratory therapist (n ⫽ 1), an exercise physiologist (n ⫽ 1), or a licensed medical herbalist (n ⫽ 1). The source of the answer was not disclosed in three of the Web sites. One of the sites charged $35 for the answer to the e-mailed question (payable by credit card).
Technologic Innovation Six of the Web sites used multimedia capability, specifically audio explanations or videos. Interactive activities were incorporated in only two of the Web sites; these were both interactive quizzes. Three sites contained material that could be personalized to the reader’s specific condition or symptoms. Support groups were included in 15 Web sites. These went by the name of chat rooms, chat sessions, pen pals, support groups, bulletin boards, message boards, and discussion groups. Eleven of these 15 sites (73%)
Table 5—No. of Core Educational Concepts Stratified by Sponsorship No. of Core Educational Concepts Sponsorships
Mean (⫾ SD)
Government Organization University Commercial Individual Unknown
5.9 (2.4) 5.2 (2.5) 4.8 (2.1) 4.7 (2.6) 3.0 (1.9) 6*
Median (Range) 6.0 (2–8) 5.0 (1–8) 4.5 (2–8) 4.5 (1–8) 3.0 (1–6)
*The “unknown” group had only one Web site with asthma education material. www.chestjournal.org
Discussion The Internet is a powerful new communication technology that has the capacity to radically affect the patient-physician relationship. Patients now have unprecedented access to health and medical information. Health-related Web sites represent one of the most common noncommercial Web sites in the world.2 Patients’ access to health-related information has important implications to common chronic diseases such as asthma. However, true benefit from asthma-related Web sites depends, in part, on the quality of the information and on accessibility to the information. Our study demonstrates the limits of currently available patient information about asthma on the Internet and confirms previously reported findings regarding communication barriers on the World Wide Web. One of the principle barriers to effective communication relates to language. Although the World Wide Web has a worldwide reach, few asthma education Web sites were multilingual. Even for English-language speakers, communication is a problem. The Web sites require a broad range of readability levels; however, the vast majority of the asthma education material is written at a reading level well above the average CHEST / 121 / 4 / APRIL, 2002
1305
patient’s reading comprehension level. Interestingly, the readability level is similar regardless of who sponsors the Web site. These communication barriers are obviously significant obstacles for using the World Wide Web for asthma patient education. A second barrier to asthma patient education on the Internet is technical in nature. In our study, Web site design varied widely, resulting in a wide variety of times required to open the first page when using a 28.800-bps modem. The median time of 33.6 s exceeded the 30 s most users are willing to wait for a page to load before trying another site. It is, therefore, likely that many of the sites that we reviewed would not be used because people would become impatient and lose interest while waiting up to 292 s for the Web site to open. This delay in loading the page is caused mainly by the incorporation of many graphics into the site that do little to enhance the content of the Web site. Although increased availability of broad-band connectivity with cable modems and direct subscriber lines will improve this problem, it is likely that a significant proportion of users around the world will continue to access the Internet using narrow-band modems. In addition to problems with accessibility, we also found significant quality issues related to asthma patient information sites. These problems related both to general issues of health-related information covered by the HONcode, and to quality issues specific to asthma information. The number of Web sites that conform to all the HON criteria is surprisingly small. The two most commonly omitted principles related to currency (identifying when the page was written) and sponsorship (who paid for or sponsored the site). Currency is critical information because once information is on the World Wide Web, it can remain there indefinitely, and it is impossible for the user to evaluate the currency of the data. This problem of currency is highlighted by the fact that we found blatantly inaccurate information on one Web site. The contact person listed did not respond to our e-mail about this inaccuracy, and the erroneous information remains on the Web site. This example highlights the need for maintenance and currency stamps on health-related Web sites. Similarly, failing to include information about the sources of funding of the sites is a potential problem. Without this information, it is difficult to evaluate the objectivity of the information on the Web site. Unfortunately, even the Web sites that displayed the HON insignia did not always conform to all the criteria, which is especially troublesome because the presence of the insignia implies that it is a Web site people can trust to present accurate information in a fair manner. However, the insignia is used on an honor system and cannot be monitored.19 1306
We also found significant problems related to the quality of information specific to asthma. Several national organizations (NAEP, American Lung Association, and AMA) have endorsed eight core asthma educational concepts for patients. The number of core educational concepts included in each of the Web sites we reviewed was variable, and only a minority of the sites contained all eight educational concepts. Sites sponsored by individuals contained the least number of educational concepts. Most of these sites are sponsored by people with asthma or by their family members and provide information about their experience with asthma. Therefore, it is understandable that they contain fewer educational concepts. Interestingly, university-sponsored Web sites did not contain more core educational concepts than government and professional organization sponsored sites. This is in contrast to likely public perception that universities are leading authorities in education and should be implicitly trusted to provide necessary information. This finding is consistent, however, with a previous study of information related to managing childhood diarrhea.20 In the study, the source of the information was unrelated to its quality, and University-sponsored sites had no higher quality than did other sponsored sites. One potential advantage offered by the Internet as an educational vehicle is the ability to actively involve the learner in using technology. Actively involving learners improves educational efficacy. Asthma education Web sites currently use little innovative technology. Few Web sites contain multimedia material, interactive activities, or support groups. We did not find any other unanticipated innovative technology. Our evaluation reveals that Web sites have not been advanced far beyond the stage of printed material. Several potential limitations to this study exist. First, given the total number of Web sites produced in the search for asthma patient information, we reviewed relatively few Web sites; however, as we reviewed Web sites, we found increased Web site duplication in the searches. The same Web sites repeatedly appeared between and even within the individual searches, resulting in diminishing returns, which suggests that the total number of sites found on the large searches is somewhat misleading. In addition, we used a search strategy to locate Web sites that is likely to be used by the general public. It is unlikely that a patient searching for information on asthma will follow the search results deeper than we did. Therefore, we are confident that the sites that most patients would visit were evaluated. We acknowledge, however, that entering other search strategies could yield a different list of Web sites. The second potential limitation relates to using the Flesch score. The Flesch score and the Flesch-Kincaid readability level are designed for general text, not specifically for medical text. Evaluating medical literature Special Report
Table 7—Web Sites Conforming to All HON Principles or Flesch-Kincaid Readability Levels < 9.0
Host Computer Address www.mdnet.de fbhc.org people.delphi.com www.vh.org www.cdc.gov www.acponline.org www.ispub.com pharminfo.com www.gweep.net www.myasthma.com www.uta.fi www.sinuses.com worldmall.com 12.4.5.33 www.nhlbisupport.com www.nhlbi.nih.gov www.lungusa.org www.asthmalearninglab.com www.meddean.luc.edu www.aafp.org www.temple.edu www.allergynet.com.au www.mdadvice.com www.aarc.org www.users.globalnet.co.uk www.aanma.org www.healthgate.com www.aarrc.com www.doctorgupta.com
HON Educational Flesch-Kincaid Principles, Concepts, Readability No. No. Level 8 8 8 8 8 8 8 8 8 8 8 8 8 8 6 6 5 4 6 5 6 6 7 6 5 7 6 7 6
5 3 3 7 6 5 1 8 3 8 1 6 4 7 8 8 8 2 8 8 6 7 8 3 4 2 8 2 3
10.6 10.0 10.5 12.0 10.6 12.0 12.0 12.0 9.7 9.8 8.9 10.0 11.9 8.8 5.2 5.2 6.3 7.1 7.1 7.4 7.6 7.9 8.0 8.0 8.2 8.2 8.7 8.8 8.8
with these tools could possibly lead to overestimated scores, given the length of most medically related words. However, because medical terms in general require a higher reading level to clearly understand, we believe that our results are relevant. We are unaware of any other standard reading instrument we could use related to medical vocabulary and readability for the general public. Our study demonstrates the considerable variability in the accessibility and quality of asthma education on the World Wide Web. The majority of sites are written in English only and at a reading level higher than the comprehension level of patients. The average site does not conform to the HONcode quality criteria and does not contain all core educational concepts. In addition, little innovative technology is being used on asthma education Web sites. It is clear that the medical and medical education community will need to contribute to health-related sites if high quality, accessibility, and educational innovation are to be realized and if the World Wide Web is to develop into an important resource for our patients. Although the authors do not endorse the use of www.chestjournal.org
particular Web sites, Table 7 contains the host computer addresses of those Web sites that conform with all HON criteria and those containing asthma education material with a Flesch-Kincaid readability level ⬍ 9.0. ACKNOWLEDGMENT: The authors thank Lori StrommerPace for her technical assistance during the evaluation of the Web sites.
References 1 Internet surveys. Available at: http. //www.nua.net/surveys/ how_many_online/index.html. Accessed March 15, 2002 2 Lawrence S, Giles CL. Accessibility of information on the Web. Nature 1999; 400:107–109 3 Spencer JA, Jordan, RK. Learner centered approaches in medical education. BMJ 1999; 318:1280 –1283 4 Kim P, Eng TR, Deering MJ, et al. Published criteria for evaluating health related Websites: review. BMJ 1999; 318: 647– 649 5 LaPerriere B, Edwards P, Romeder JM. Using the Internet to support self-care. Can Nurse 1998; 94:47– 48 6 Graber MA, Roller CM, Kaeble B. Readability levels of patient education material on the World Wide Web. J Fam Pract 1999; 48:58 – 61 7 Peterson MW. Are medical Websites a boon or bane for patients and physicians? Med Behav 1999:33–34 8 Cote J, Cartier A, Robichaud P, et al. Influence on asthma morbidity of asthma education programs based on selfmanagement plans following treatment optimization. Am J Respir Crit Care Med 1997; 155:1509 –1514 9 de Oliveira MA, Bruno VF, Ballini LS, et al. Evaluation of an educational program for asthma control in adults. J Asthma 1997; 34:395– 403 10 van der Palen J, Klein JJ, Zielhuis GA, et al. The role of self-treatment guidelines in self-management education for adult asthmatics. Respir Med 1998; 92:668 – 675 11 Gallefoss F, Bakke PS, Rsgaard PK. Quality of life assessment after patient education in a randomized controlled study on asthma and chronic obstructive pulmonary disease. Am J Respir Crit Care Med 1999; 159:812– 817 12 Liljas B, Lahdensuo A. Is asthma self-management cost effective? Patient Educ Couns 1997; 32:S97–S104 13 National Asthma Education and Prevention Program. Expert panel report 2: guidelines for the diagnosis and management of asthma. Bethesda, MD: National Institutes of Health, April 1997; Publication No. 97– 4051 14 Peterson MW, Strommer-Pace L, Dayton C. Asthma patient education: current utilization in pulmonary training programs. J Asthma 2001; 38:261–267 15 Eagle Tech Writing Service. Available at: http: //rnrs.com/ eagle. Accessed March 15, 2002 16 American Lung Association. General asthma information. Available at: http://www.lungusa.org/asthma/index.html. Accessed March 15, 2002 17 Journal of the American Medical Association. Asthma Information Center. Available at: http://www.ama-assn.org/special/ asthma. Accessed February 2001; this site no longer exists 18 Health on the Net Code of Conduct, version 1.6. Available at: http://www.hon.ch/HONcode/HONcode_check.html. Accessed March 15, 2002 19 Eysenbach G. Towards ethical guidelines for e-health. J Med Internet Res 2000; 2:e7 20 McClung HJ, Murray RD, Heitlinger LA. The Internet as a source of current patient information. Pediatrics 1998; 101:E2 CHEST / 121 / 4 / APRIL, 2002
1307
Study objectives: To measure the accessibility and quality of currently available asthma education World Wide Web sites using the following criteria: accessibility by readability, language, and download time; information quality based on inclusion of core educational concepts and compliance with Health On the Net (HON) principles; and utilization of innovative technology. Design: Objective evaluation of 145 Web sites. Measurements and results: Four search engines or directories (Yahoo, HON, Alta Vista, and Healthfinder) were searched for “asthma, patient information.” A maximum of 50 Web sites from each search engine or directory was evaluated. Only 90 of the 145 Web sites actually contained asthma educational material. The mean (ⴞ SD) time necessary to open each Web site on a 28.800-bits-per-second modem was 33.6 ( ⴞ 36.6) s. The mean number of graphics on the Web sites was 24.6 ( ⴞ 30.2) files per page. The educational material required a mean reading level beyond the 10th grade. Only nine Web sites contained multilingual asthma education material. The mean number of HON principles with which the Web sites conformed was 6.3 ( ⴞ 1.0) of 8 principles; 14 Web sites conformed to all the HON criteria. The average Web site contained 4.9 (ⴞ 2.5) of 8 core asthma educational concepts, and only 20 Web sites contained all 8 educational concepts. Very few Web sites utilized innovative educational technology. Conclusions: While patient asthma education Web sites are common, asthma educational material contains many accessibility barriers, is highly variable in quality and content, and takes little innovative use of technology. Patient educational material currently available on the World Wide Web fails to meet the information needs of patients. (CHEST 2002; 121:1301–1307) Key words: asthma; information distribution; Internet; language; learning; lung disease; patient education; reading; technology; World Wide Web Abbreviations: AMA ⫽ American Medical Association; bps ⫽ bits per second; HON ⫽ Health On the Net; HONcode ⫽ Health On the Net Code of Conduct; NAEP ⫽ National Asthma Education and Prevention Program
of the World Wide Web compoT henentintroduction of the Internet has made information available to people on an unprecedented scale. It is estimated that 360 million people worldwide cur*From the Department of Internal Medicine, University of Iowa College of Medicine, Iowa City, IA. Dr. Croft is currently an Epidemic Intelligence Service Officer with the Centers for Disease Control and Prevention, Atlanta, GA. This study did not receive financial support from any company or organization. The opinions contained herein are the views of the authors and are not meant to represent the views of any agency or institution. Manuscript received April 9, 2001; revision accepted October 30, 2001. Correspondence to: Donita R. Croft, MD, MS, PO Box 2659, Madison, WI 53701; e-mail: [email protected] www.chestjournal.org
rently have access to the World Wide Web.1 The World Wide Web contains about 800 million publicly available pages, and approximately 3% are health-related Web sites.2 The World Wide Web is For editorial comment see page 1015 potentially a very powerful vehicle for providing patient education. It has several advantages as a health-related educational tool for patients. First, it is relatively inexpensive to disseminate information to millions of people, even across national borders. Second, the World Wide Web can be used to link people with similar conditions, which is especially CHEST / 121 / 4 / APRIL, 2002
1301
important for people who are homebound, live in geographically remote areas, or have rare conditions. Third, the design and structure of the Internet make it difficult to disrupt transmission of information. Finally, computer software allows for significant interactivity with the user and provides the opportunity for users to be actively engaged in learning. Modern adult learning theory suggests that active participation by the learner significantly enhances the learning.3 However, the World Wide Web also presents unique new challenges as an effective patient educational vehicle. The very structure of the Internet leads to a very decentralized control. It is therefore difficult to enforce quality-control measures to ensure that the information provided is accurate. Peer review has been traditionally used for quality control of medical publications; however, this is much more problematic on the World Wide Web. Because of the ease with which any individual can launch a Web site, inaccurate or even deceptive material can be transmitted to vast numbers of people before it is identified as such. It is difficult for patients and professionals alike to judge the quality of information. Several organizations, institutions, and journals have developed generic rating tools in an attempt to objectively evaluate medical information on the World Wide Web. The most commonly used principles in these rating tools are disclosure of the authors and sponsors, information currency, accessibility, and ease of use of the information. The American Medical Association (AMA), British Healthcare Internet Association, and Health On the Net (HON) Foundation are examples of groups that have developed such tools.4 In addition to quality control, accessibility is a potential problem on Web-based educational sites. Accessibility can be viewed both in technical terms and in communication terms. In technical terms, this is often referred to as the “last mile” problem because most World Wide Web users currently access the system using modems with limited communication capacity. Since users have a limited capacity for waiting for information, poorly designed pages with extensive graphics present a barrier for delivering information effectively. Another limitation is communication barriers. The majority of information on the World Wide Web is written in English only, which limits patient access given the worldwide reach of the media.5 Even for English-speaking users, the readability level of the information can be a barrier. In evaluating 50 healthrelated Web sites, Graber et al6 found the average reading level at about a 10th-grade level despite the fact that the level of reading comprehension is considerably lower in most patient populations. 1302
The World Wide Web thus presents both opportunities and challenges for patient education. Patient self-management, however, is an important component of chronic disease management. It is widely believed that a patient who is educated about his or her health-related problem will better adhere to treatment plans and have improved health outcomes. Patients also want information about their own health and management. It has been estimated that between 30% and 50% of Internet users search for health-related information on the World Wide Web on a regular basis.7 In pulmonary medicine, we deal with several chronic diseases in our patient population. Asthma is probably the best-studied pulmonary disease model in which patient self-management is incorporated. Several studies8 –11 have reported that asthma patient education programs improve compliance with medications, decrease emergency department visits and hospitalizations, and improve quality of life. Formal asthma education programs have also been shown to be cost-effective.12 Recognizing this, patient education is an integral component in the National Asthma Education and Prevention Program (NAEP) Expert Panel Report 2.13 However, formal asthma education programs are costly, time-consuming, and require significant expertise. Consequently, even many training programs in pulmonary medicine do not incorporate formal patient education programs into their asthma care.14 The World Wide Web presents itself as a potential tool for solving this problem; indeed, many asthma patient education sites are now available on the World Wide Web. Given the challenges presented by the World Wide Web as a quality education tool, however, we know very little about the nature of the patient educational materials available on asthma management. We undertook this study to measure the quality of currently available Internet sites using accessibility, information quality, and innovative design as our criteria. Based on previous studies of both patient and professional information offered on the Internet, we postulated that asthma patient information would be highly variable in each of these domains. Materials and Methods Four search engines or directories (Yahoo, Alta Vista, Healthfinder and HON) were chosen as the sources of the Internet sites. These were chosen to provide a diverse variety of healthrelated and non-health-related search engines and directories. Using the key words “asthma” and “patient information” or “asthma, patient information,” a search for Web sites was undertaken in each search engine or directory. The total number of Web sites produced by the searches was recorded. The first 50 Web sites from each search engine or directory were consecuSpecial Report
tively evaluated. If ⬍ 50 Web sites were produced by the search, all sites were evaluated. Web sites evaluated in one search were excluded from subsequent search result evaluations. Sites that did not include actual asthma patient information were also excluded. The accessibility of information on the Web sites was assessed by several different means. As a measure of the time necessary to access each Web site, FrontPage 2000 (Microsoft; Redmond, WA) was used to determine the amount of time required to load the homepage of the Web site using a 28.800-bits-per-second (bps) modem. In addition, because the number of graphics in a Web site is directly related to the amount of time needed to open the site, the graphics in the Web sites were quantified. This was accomplished by opening each site in its hypertext markup language source and counting the number of .gif and .jpg files (the most common file extensions for graphical files) included on the home page. The readability of each site was determined by copying the asthma education material from the site, pasting it into a Microsoft Word 97 (Microsoft) document, and obtaining the Flesch score and Flesch-Kincaid readability grade level. The Flesch score ranges from 0 to 100, from the most difficult to the easiest to read, respectively.6 The Flesch-Kincaid readability grade level determines the difficulty of the material by measuring the length of the words and sentences and converting the results into a reading level from third to 12th grade.15 The FleschKincaid readability grade level is unable to discriminate material less complex than the third grade or more complicated than the 12th grade. As a final means of determining the accessibility of information on the Web sites, the language used in each site was reviewed; if at least some of the educational materials were in more than one language, the Web site was identified as multilingual. Eight core educational concepts were identified by comparing the asthma education material of the NAEP,13 the American Lung Association,16 and the AMA.17 The eight asthma educational concepts that the three groups had in common were used as core educational concepts. These concepts are listed in Table 1. The educational material on the searched Web sites was evaluated, and inclusion of any of the eight concepts was quantified. The use of innovative technology on the Web sites was evaluated by reviewing the sites and recording the use of multimedia capability, interactive activities, personalizable material, support groups, and presence of synchronous or asynchronous communication to answer specific questions posed by the user. The use of multimedia capability was defined as the application of video or audio media for the purpose of teaching about asthma. The Web site was recorded as being interactive if it contained activities in which the responses varied, depending on the information provided by the user. These activities included tests or quizzes in which responses were tailored to the provided answers. Information that changed in response to data
Table 1—Core Asthma Educational Concepts Used to Evaluate Content of Asthma Patient Education Web Sites 1. 2. 3. 4. 5. 6. 7. 8.
Basic asthma pathophysiology Triggers of asthma attacks Avoidance of asthma triggers Mechanism of action of medications for asthma Importance of use of anti-inflammatory medications Importance of availability of rescue medications Self-assessment of respiratory status Necessity of an action plan dependent on respiratory status
www.chestjournal.org
specific to the reader’s condition or symptoms was considered personalizable material. An example of personalizable material is recommendations on the use of metered-dose inhalers based on peak flow readings. Support groups were defined as a mechanism by which the reader could be in contact with other users of the Web site. The Web sites were also reviewed for the capacity to answer asthma-related questions asked by the reader via e-mail or forms. The disclosure of the credentials of the person answering the questions was noted. Also, the sponsorship of the site was recorded in the following categories: commercial, government, university, organization, individual, or unknown. This information was obtained from sponsorship information, Web site content or, if not obvious in the site, from the address of the Web site. For example, if the address ended in “.com,” it was considered a commercial site. The quality of content was evaluated using the Site-Checker of the HON Code of Conduct (HONcode).18 This quality rating tool was used because the HONcode is comprehensive and the Site-Checker is easily navigated. The HONcode requires the identity and credentials of the author, adherence to confidentiality, timeliness of modifications, and disclosure of the funding source of the Web sites. The HONcode principles are listed in Table 2. The presence or absence of the HON insignia was recorded. The HON insignia is available for placement on Web
Table 2—HONcode Principles* 1. Any medical/health advice provided and hosted on this site will only be given by medically or health-trained and qualified professionals unless a clear statement is made that a piece of advice offered is from a nonmedically/health qualified individual/organization. 2. The information provided on this site is designed to support, not replace, the relationship that exists between a patient/site visitor and his/her existing physician. 3. Confidentiality of data relating to individual patients and visitors to a medical/health Web site, including their identity, is respected by this Web site. The Web site owners undertake to honor or exceed the legal requirements of medical/health information privacy that apply in the country and state where the Web site and mirror sites are located. 4. Where appropriate, information contained on this site will be supported by clear references to source data and, where possible, have specific HTML links to that data. The date when a clinical page was last modified will be clearly displayed (eg, at the bottom of the page). 5. Any claims relating to the benefits/performance of a specific treatment, commercial product or service will be supported by appropriate, balanced evidence in the manner outlined above in principle 4. 6. The designers of this Web site will seek to provide information in the clearest possible manner and provide contact addresses for visitors who seek further information or support. The Webmaster will display his/her e-mail address clearly throughout the Web site. 7. Support for this Web site will be clearly identified, including the identities of commercial and noncommercial organizations that have contributed funding, services or material for the site. 8. If advertising is a source of funding it will be clearly stated. A brief description of the advertising policy adopted by the Web site owners will be displayed on the site. Advertising and other promotional material will be presented to viewers in a manner and context that facilitates differentiation between it and the original material created by the institution operating the site. *HTML ⫽ hypertext markup language. CHEST / 121 / 4 / APRIL, 2002
1303
sites based on an honor system and is meant to indicate that the site adheres to all of the HON principles. In addition, obviously inaccurate information on the Web sites was noted. The data were recorded and analyzed using Microsoft Access 97 (Microsoft).
Results Between November 1999 and February 2000, we identified and evaluated 145 Web sites. The search of HON revealed 950 Web sites; Yahoo revealed 30 Web sites; Alta Vista revealed 245,630 Web sites; and Healthfinder revealed 87 Web sites. From these groups, 50 Web sites were evaluated from HON, 30 from Yahoo, 50 from Alta Vista, and 15 from Healthfinder. We evaluated ⬍ 50 sites from Healthfinder because they represented duplicate Web sites that we evaluated using other search engines. Table 3 displays characteristics of the Web sites. Five Web sites were inaccessible either because they were technically unavailable at the time of the search (n ⫽ 3) or the sites were not written in English (n ⫽ 2). Of the 140 accessible Web sites, only 90 sites (64.3%) actually contained asthma education material. The Web sites that did not contain asthma education material contained information about product sales, research recruitment, obtaining an appointment in a doctor’s practice, or conference schedules, or the sites were simply portals to other Web sites containing asthma education material. Accessibility The mean time required to open the Web sites on a 28.800-bps modem, as determined by FrontPage 2000, was 33.6 (⫾ 36.6) s, and the median was 25 s (range, 0 to 292 s). This result was based on 112 Web sites because 33 of the Web sites would not allow
Table 3—Web Site Characteristics Characteristics Accessibility Accessible during search Inaccessible during search Could not read language of the Web site Asthma education material Contained Not contained Sponsorship Commercial Organization University Government Individual Unknown
1304
FrontPage 2000 to open them in an editor. The mean number of graphics located on the Web sites was 24.6 ( ⫾ 30.2) files per page, and the median was 16 files per page (range, 0 to 206 files per page). The educational material had a mean Flesch score of 46.3 ( ⫾ 13.5) and a median of 47.6 (range, 8.9 to 75.7). The mean Flesch-Kincaid readability level of these Web sites was 10.3 (or 10th grade, third month) [⫾ 1.7], and the median was 10.5 (range, 5.2 to 12.0.) When stratified by sponsorship, the mean FleschKincaid readability level ranged from 10.2 (company and organization) to 10.9 (individual). Twenty-seven of the Web sites scored at the maximum readability, 12.0. Only nine Web sites contained multilingual asthma education material. These Web sites were written in Spanish and English (n ⫽ 2); French and English (n ⫽ 2); Italian, French, and English (n ⫽ 1); Spanish, French and English (n ⫽ 1); German and English (n ⫽ 1); Korean and English (n ⫽ 1); and Finnish and English (n ⫽ 1). Of the Web sites that were written in a single non-English language, one was in Spanish only and one in Turkish only. Quality Control We first quantified generic quality by determining the adherence of the Web site to HONcode principles. The mean number of HON principles followed in the Web sites with asthma education material was 6.3 (⫾ 1.0) principles, and the median was 6 principles (range, 4 to 8 principles). Table 4 shows the number of Web sites conforming to each of the HON principles. Only 14 of the Web sites conformed to all eight of the HON criteria. The HON insignia was present in 16 of the Web sites containing asthma education material. Of these 16 Web sites, only 6 sites conformed to all of the HON principles. One Web site sponsored by a university erroneously stated that Serevent (GlaxoSmithKline; Research Triangle Park, NC) is a quick-acting medi-
No. of Web Sites 140 3 2 90 50 67 36 22 10 8 2
Table 4 —Web Sites Conforming to the HON Principles Principle No.
No. of Web Sites (%)
1 2 3 4 5 6 7 8 All 8
79 (87.8) 90 (100.0) 90 (100.0) 24 (26.7) 89 (98.9) 73 (81.1) 34 (37.8) 87 (96.7) 14 (15.6)
Special Report
cine. The investigators e-mailed the two contacts listed on the Web site about this incorrect information. One e-mail was returned undeliverable, and no response was received from the second contact. The misinformation remained on the Web site several months later. Five of the Web sites were based entirely on practices of complementary medicine, whereas many sites mentioned alternative medicine practices. We next evaluated the sites based on whether they provided the eight core educational components. Of the Web sites containing asthma education material, the mean number of core educational concepts was 4.9 (⫾ 2.5) out of 8, and the median was 5 concepts (range, 1 to 8 concepts). The number of concepts included in the Web sites, when stratified by sponsorship of the Web site, is displayed in Table 5. The government-sponsored sites contained the most core concepts, and the individual-sponsored sites contained the least core educational concepts. Twenty Web sites contained all eight core educational concepts. These were sponsored by organizations (n ⫽ 8), commercial groups (n ⫽ 7), government (n ⫽ 4), and a university (n ⫽ 1). Table 6 lists the core educational concepts and their frequency of inclusion in the Web sites. The most frequently included educational concept was triggers of asthma attacks (90.1%), and the least commonly included concept was the importance of the availability of “rescue” medications (37.4%).
Table 6 —Frequency of Inclusion of Core Educational Concepts Educational Concepts Basic asthma pathophysiology Triggers of asthma attacks Avoidance of asthma triggers Mechanism of action of medications for asthma Importance of use of anti-inflammatory medications Importance of availability of rescue medications Self-assessment of respiratory status Necessity of an action plan dependent on respiratory status
No. of Websites (%) 76 (84.4) 82 (91.1) 65 (72.2) 54 (60.0) 50 (55.6) 34 (37.8) 42 (46.7) 39 (43.3)
were commercially sponsored. In 20 of the Web sites, the reader had the capacity to e-mail an asthma-related question. These Web sites reported that the questions were answered by a physician (n ⫽ 11), a physician or a pharmacist (n ⫽ 1), a registered nurse (n ⫽ 1), a nurse practitioner (n ⫽ 1), a respiratory therapist (n ⫽ 1), an exercise physiologist (n ⫽ 1), or a licensed medical herbalist (n ⫽ 1). The source of the answer was not disclosed in three of the Web sites. One of the sites charged $35 for the answer to the e-mailed question (payable by credit card).
Technologic Innovation Six of the Web sites used multimedia capability, specifically audio explanations or videos. Interactive activities were incorporated in only two of the Web sites; these were both interactive quizzes. Three sites contained material that could be personalized to the reader’s specific condition or symptoms. Support groups were included in 15 Web sites. These went by the name of chat rooms, chat sessions, pen pals, support groups, bulletin boards, message boards, and discussion groups. Eleven of these 15 sites (73%)
Table 5—No. of Core Educational Concepts Stratified by Sponsorship No. of Core Educational Concepts Sponsorships
Mean (⫾ SD)
Government Organization University Commercial Individual Unknown
5.9 (2.4) 5.2 (2.5) 4.8 (2.1) 4.7 (2.6) 3.0 (1.9) 6*
Median (Range) 6.0 (2–8) 5.0 (1–8) 4.5 (2–8) 4.5 (1–8) 3.0 (1–6)
*The “unknown” group had only one Web site with asthma education material. www.chestjournal.org
Discussion The Internet is a powerful new communication technology that has the capacity to radically affect the patient-physician relationship. Patients now have unprecedented access to health and medical information. Health-related Web sites represent one of the most common noncommercial Web sites in the world.2 Patients’ access to health-related information has important implications to common chronic diseases such as asthma. However, true benefit from asthma-related Web sites depends, in part, on the quality of the information and on accessibility to the information. Our study demonstrates the limits of currently available patient information about asthma on the Internet and confirms previously reported findings regarding communication barriers on the World Wide Web. One of the principle barriers to effective communication relates to language. Although the World Wide Web has a worldwide reach, few asthma education Web sites were multilingual. Even for English-language speakers, communication is a problem. The Web sites require a broad range of readability levels; however, the vast majority of the asthma education material is written at a reading level well above the average CHEST / 121 / 4 / APRIL, 2002
1305
patient’s reading comprehension level. Interestingly, the readability level is similar regardless of who sponsors the Web site. These communication barriers are obviously significant obstacles for using the World Wide Web for asthma patient education. A second barrier to asthma patient education on the Internet is technical in nature. In our study, Web site design varied widely, resulting in a wide variety of times required to open the first page when using a 28.800-bps modem. The median time of 33.6 s exceeded the 30 s most users are willing to wait for a page to load before trying another site. It is, therefore, likely that many of the sites that we reviewed would not be used because people would become impatient and lose interest while waiting up to 292 s for the Web site to open. This delay in loading the page is caused mainly by the incorporation of many graphics into the site that do little to enhance the content of the Web site. Although increased availability of broad-band connectivity with cable modems and direct subscriber lines will improve this problem, it is likely that a significant proportion of users around the world will continue to access the Internet using narrow-band modems. In addition to problems with accessibility, we also found significant quality issues related to asthma patient information sites. These problems related both to general issues of health-related information covered by the HONcode, and to quality issues specific to asthma information. The number of Web sites that conform to all the HON criteria is surprisingly small. The two most commonly omitted principles related to currency (identifying when the page was written) and sponsorship (who paid for or sponsored the site). Currency is critical information because once information is on the World Wide Web, it can remain there indefinitely, and it is impossible for the user to evaluate the currency of the data. This problem of currency is highlighted by the fact that we found blatantly inaccurate information on one Web site. The contact person listed did not respond to our e-mail about this inaccuracy, and the erroneous information remains on the Web site. This example highlights the need for maintenance and currency stamps on health-related Web sites. Similarly, failing to include information about the sources of funding of the sites is a potential problem. Without this information, it is difficult to evaluate the objectivity of the information on the Web site. Unfortunately, even the Web sites that displayed the HON insignia did not always conform to all the criteria, which is especially troublesome because the presence of the insignia implies that it is a Web site people can trust to present accurate information in a fair manner. However, the insignia is used on an honor system and cannot be monitored.19 1306
We also found significant problems related to the quality of information specific to asthma. Several national organizations (NAEP, American Lung Association, and AMA) have endorsed eight core asthma educational concepts for patients. The number of core educational concepts included in each of the Web sites we reviewed was variable, and only a minority of the sites contained all eight educational concepts. Sites sponsored by individuals contained the least number of educational concepts. Most of these sites are sponsored by people with asthma or by their family members and provide information about their experience with asthma. Therefore, it is understandable that they contain fewer educational concepts. Interestingly, university-sponsored Web sites did not contain more core educational concepts than government and professional organization sponsored sites. This is in contrast to likely public perception that universities are leading authorities in education and should be implicitly trusted to provide necessary information. This finding is consistent, however, with a previous study of information related to managing childhood diarrhea.20 In the study, the source of the information was unrelated to its quality, and University-sponsored sites had no higher quality than did other sponsored sites. One potential advantage offered by the Internet as an educational vehicle is the ability to actively involve the learner in using technology. Actively involving learners improves educational efficacy. Asthma education Web sites currently use little innovative technology. Few Web sites contain multimedia material, interactive activities, or support groups. We did not find any other unanticipated innovative technology. Our evaluation reveals that Web sites have not been advanced far beyond the stage of printed material. Several potential limitations to this study exist. First, given the total number of Web sites produced in the search for asthma patient information, we reviewed relatively few Web sites; however, as we reviewed Web sites, we found increased Web site duplication in the searches. The same Web sites repeatedly appeared between and even within the individual searches, resulting in diminishing returns, which suggests that the total number of sites found on the large searches is somewhat misleading. In addition, we used a search strategy to locate Web sites that is likely to be used by the general public. It is unlikely that a patient searching for information on asthma will follow the search results deeper than we did. Therefore, we are confident that the sites that most patients would visit were evaluated. We acknowledge, however, that entering other search strategies could yield a different list of Web sites. The second potential limitation relates to using the Flesch score. The Flesch score and the Flesch-Kincaid readability level are designed for general text, not specifically for medical text. Evaluating medical literature Special Report
Table 7—Web Sites Conforming to All HON Principles or Flesch-Kincaid Readability Levels < 9.0
Host Computer Address www.mdnet.de fbhc.org people.delphi.com www.vh.org www.cdc.gov www.acponline.org www.ispub.com pharminfo.com www.gweep.net www.myasthma.com www.uta.fi www.sinuses.com worldmall.com 12.4.5.33 www.nhlbisupport.com www.nhlbi.nih.gov www.lungusa.org www.asthmalearninglab.com www.meddean.luc.edu www.aafp.org www.temple.edu www.allergynet.com.au www.mdadvice.com www.aarc.org www.users.globalnet.co.uk www.aanma.org www.healthgate.com www.aarrc.com www.doctorgupta.com
HON Educational Flesch-Kincaid Principles, Concepts, Readability No. No. Level 8 8 8 8 8 8 8 8 8 8 8 8 8 8 6 6 5 4 6 5 6 6 7 6 5 7 6 7 6
5 3 3 7 6 5 1 8 3 8 1 6 4 7 8 8 8 2 8 8 6 7 8 3 4 2 8 2 3
10.6 10.0 10.5 12.0 10.6 12.0 12.0 12.0 9.7 9.8 8.9 10.0 11.9 8.8 5.2 5.2 6.3 7.1 7.1 7.4 7.6 7.9 8.0 8.0 8.2 8.2 8.7 8.8 8.8
with these tools could possibly lead to overestimated scores, given the length of most medically related words. However, because medical terms in general require a higher reading level to clearly understand, we believe that our results are relevant. We are unaware of any other standard reading instrument we could use related to medical vocabulary and readability for the general public. Our study demonstrates the considerable variability in the accessibility and quality of asthma education on the World Wide Web. The majority of sites are written in English only and at a reading level higher than the comprehension level of patients. The average site does not conform to the HONcode quality criteria and does not contain all core educational concepts. In addition, little innovative technology is being used on asthma education Web sites. It is clear that the medical and medical education community will need to contribute to health-related sites if high quality, accessibility, and educational innovation are to be realized and if the World Wide Web is to develop into an important resource for our patients. Although the authors do not endorse the use of www.chestjournal.org
particular Web sites, Table 7 contains the host computer addresses of those Web sites that conform with all HON criteria and those containing asthma education material with a Flesch-Kincaid readability level ⬍ 9.0. ACKNOWLEDGMENT: The authors thank Lori StrommerPace for her technical assistance during the evaluation of the Web sites.
References 1 Internet surveys. Available at: http. //www.nua.net/surveys/ how_many_online/index.html. Accessed March 15, 2002 2 Lawrence S, Giles CL. Accessibility of information on the Web. Nature 1999; 400:107–109 3 Spencer JA, Jordan, RK. Learner centered approaches in medical education. BMJ 1999; 318:1280 –1283 4 Kim P, Eng TR, Deering MJ, et al. Published criteria for evaluating health related Websites: review. BMJ 1999; 318: 647– 649 5 LaPerriere B, Edwards P, Romeder JM. Using the Internet to support self-care. Can Nurse 1998; 94:47– 48 6 Graber MA, Roller CM, Kaeble B. Readability levels of patient education material on the World Wide Web. J Fam Pract 1999; 48:58 – 61 7 Peterson MW. Are medical Websites a boon or bane for patients and physicians? Med Behav 1999:33–34 8 Cote J, Cartier A, Robichaud P, et al. Influence on asthma morbidity of asthma education programs based on selfmanagement plans following treatment optimization. Am J Respir Crit Care Med 1997; 155:1509 –1514 9 de Oliveira MA, Bruno VF, Ballini LS, et al. Evaluation of an educational program for asthma control in adults. J Asthma 1997; 34:395– 403 10 van der Palen J, Klein JJ, Zielhuis GA, et al. The role of self-treatment guidelines in self-management education for adult asthmatics. Respir Med 1998; 92:668 – 675 11 Gallefoss F, Bakke PS, Rsgaard PK. Quality of life assessment after patient education in a randomized controlled study on asthma and chronic obstructive pulmonary disease. Am J Respir Crit Care Med 1999; 159:812– 817 12 Liljas B, Lahdensuo A. Is asthma self-management cost effective? Patient Educ Couns 1997; 32:S97–S104 13 National Asthma Education and Prevention Program. Expert panel report 2: guidelines for the diagnosis and management of asthma. Bethesda, MD: National Institutes of Health, April 1997; Publication No. 97– 4051 14 Peterson MW, Strommer-Pace L, Dayton C. Asthma patient education: current utilization in pulmonary training programs. J Asthma 2001; 38:261–267 15 Eagle Tech Writing Service. Available at: http: //rnrs.com/ eagle. Accessed March 15, 2002 16 American Lung Association. General asthma information. Available at: http://www.lungusa.org/asthma/index.html. Accessed March 15, 2002 17 Journal of the American Medical Association. Asthma Information Center. Available at: http://www.ama-assn.org/special/ asthma. Accessed February 2001; this site no longer exists 18 Health on the Net Code of Conduct, version 1.6. Available at: http://www.hon.ch/HONcode/HONcode_check.html. Accessed March 15, 2002 19 Eysenbach G. Towards ethical guidelines for e-health. J Med Internet Res 2000; 2:e7 20 McClung HJ, Murray RD, Heitlinger LA. The Internet as a source of current patient information. Pediatrics 1998; 101:E2 CHEST / 121 / 4 / APRIL, 2002
1307