- Email: [email protected]
Utility of Web-based assessment of patient satisfaction with endoscopy
THE AMERICAN JOURNAL OF GASTROENTEROLOGY © 2003 by Am. Coll. of Gastroenterology Published by Elsevier Inc.
Vol. 98, No. 5, 2003 ISSN 0002-9270/03/$30.00 doi:10.1016/S0002-9270(03)00104-7
Utility of Web-Based Assessment of Patient Satisfaction With Endoscopy Gavin C. Harewood, M.D., M.Sc., Maurits J. Wiersema, M.D., and Piet C. de Groen, M.D. Division of Gastroenterology and Hepatology, Mayo Clinic, Rochester, Minnesota
OBJECTIVES: The dramatic growth of the World Wide Web (Web) holds potential for use in survey distribution and submission. Its use has not previously been studied in the context of patient satisfaction with endoscopy procedures. In this study we compared standard mail, telephone, and Web-based modes of endoscopy satisfaction survey administration with respect to response rate and response content. METHODS: An endoscopy satisfaction questionnaire consisting of seven core items from the modified Group Health Association of America (GHAA-9) was distributed to patients after routine outpatient endoscopy. Patients were randomized to receive the questionnaire by standard mail, telephone, or Web (if applicable). Response rates and satisfaction scores in the groups were compared. The nonresponders to the standard mail and Web surveys were subsequently contacted by telephone to determine their level of satisfaction. RESULTS: Response rates to the telephone survey (81% among those designated as non–Web users and 78% among Web users) and standard mail (75% non–Web users, 67% Web users) were higher than response rate to the Web-based survey (34%) (p ⱕ 0.0005). There was no significant difference in satisfaction scores among the groups or between satisfaction of nonresponders and responders. CONCLUSIONS: The potential of our Web-based survey was limited by poor response rates. This arose from privacy protection precautions that complicated access to the Web site. Future Web-based strategies for surveying patients will need to be more user-friendly while maintaining the intent of the recent Health Insurance Portability and Accountability Act regulations. (Am J Gastroenterol 2003;98:1016 –1021. © 2003 by Am. Coll. of Gastroenterology)
INTRODUCTION The rapid permeation of new telecommunications technologies throughout society has seen the emergence of the World Wide Web (Web) as an increasingly pervasive means of communication. Throughout the 1990s, because of its relative simplicity and effectiveness, there has been explosive growth of the Internet and Web as tools for seeking and communicating health and medical information (1). Increasing numbers of physicians and health care institutions are
maintaining Web sites. Neill et al. found that 31% of patients in a family medicine practice had their primary physician’s electronic mail (e-mail) address, 37% corresponded with their practitioner, and 90% of this communication was related to a medical condition (2). In recent years, there has been a rapid growth in the literature addressing the role of the Internet in medicine. The applications of these modalities are evident in patient education (3, 4), patient support networks (5), patient follow-up (6 – 8), communication among providers (3, 4, 9 –11), and communication between providers and patients (2, 12–20). On an expanding scale, the medical education community has made efforts to introduce an electronic mail system for instruction (21). Although many medical information providers run Web pages, there have been few studies evaluating the use of the Internet as a mode of survey administration. Surveys conducted by traditional means, such as the mail and telephone, are commonly used to obtain information from large groups of people. However, these methods can be time-consuming and expensive and can yield low response rates (22, 23). Considerable delay can occur between the mailing of surveys and the receipt of responses. Receipt of surveys by respondents can be difficult to confirm. Respondents often find that filling out detailed questionnaires by hand is time consuming and that returning them may require several steps. Telephone calls may occur at inconvenient times and may require an immediate response, not allowing respondents time to carefully consider the questions. The use of a Web-based mode of survey administration holds potential to greatly facilitate patient satisfaction measurement. Although the use of Web-based surveys has been previously described (24), the application of this tool in gastroenterology—specifically, in endoscopy— has not been maximized. In clinical settings such as endoscopy, patient satisfaction is being increasingly used to monitor the quality of clinical care. For patients, satisfaction represents, at least in theory, an evaluation of the health care experience based on their own values, perceptions, and interactions with the health care environment (25). The primary goals of this study were: 1) to assess the feasibility of Web-based administration of endoscopy patient satisfaction surveys; 2) to compare response rates for this method with standard mail and telephone-based questionnaires; and 3) to compare
AJG – May, 2003
Web-Based Assessment of Patient Satisfaction
1017
Table 1. Modified GHAA-9 (Core Items): ASGE Patient Satisfaction Questionnaire In terms of your satisfaction, how would you rate each of the following: 1. How long you waited to get an appointment. Excellent Very good Good Fair 2. Length of time spent waiting at the office for the procedure. Excellent Very good Good Fair 3. The personal manner (courtesy, respect, sensitivity, friendliness) of the physician who performed your procedure. Excellent Very good Good Fair 4. The technical skills (thoroughness, carefulness, competence) of the physician who performed your procedure. Excellent Very good Good Fair 5. The personal manner (courtesy, respect, sensitivity, friendliness) of the nurses and the other support staff. Excellent Very good Good Fair 6. Adequacy of explanation of what was done for you—all your questions answered. Excellent Very good Good Fair 7. Overall rating of the visit Excellent Very good Good Fair
the satisfaction scores of immediate survey responders to the scores of nonresponders who were subsequently contacted.
MATERIALS AND METHODS Study Design and Methods The study was conducted in the outpatient endoscopy unit of Rochester Methodist Hospital, Mayo Medical Center (Rochester, MN). Approximately 15% of Mayo patients are residents of the city of Rochester or surrounding Olmsted County (26). Approximately 50% of Mayo patients live within 120 miles of Rochester. Thus, Mayo Medical Center provides primary, secondary, and tertiary level care. The Division of Gastroenterology and Hepatology offers open access esophagogastroduodenoscopy (EGD) and colonoscopy. Physicians from Mayo or outside referring physicians may order these procedures without prior consultation with or approval by a staff gastroenterologist. The study was reviewed and approved by the Mayo Clinic Institutional Review Board. All patients who were referred for routine outpatient EGD or colonoscopy and who had access to a telephone were potential study candidates. None of the investigators were involved in performing these procedures, and endoscopists were blinded to patient enrollment. Patients were recruited between June 2001 and December 2001. Exclusion criteria were patient refusal to participate or inability to give verbal consent. Importantly, patients were considered eligible for enrollment irrespective of their access to the Internet; lack of Web access resulted in randomization to one of two study arms, rather than one of three study arms. By stratifying patients into two study groups (“non–Web users” and “Web users”), this permitted us to compare the demographics, response rates, and satisfaction ratings of non–Web users and Web users and thereby to make assumptions about similarities and differences between the two populations. After enrollment, patients were classified as Web users or non–Web users on the basis of Web access. Access to the
Poor Poor Poor Poor Poor Poor Poor
Web was arbitrarily defined as possession of an account, either at home or at work, with use of this account at least once every 3 days. The questionnaire distribution strategy (mail, telephone, or e-mail) was written on cards and placed in separate sealed, opaque envelopes. Immediately before the endoscopy procedure, the study coordinator chose an envelope and thereby randomly assigned each patient to receive the questionnaire by standard mail, telephone, or e-mail. Patients were informed at the time of enrollment, before endoscopy, that they would receive a questionnaire by mail or telephone within 1 wk of their procedure or, alternatively, would be required to access the survey on a Web page if randomized to the Web arm of the study. To guard against the effect of retrograde amnesia induced by sedation, patients were also reminded of the study instructions after their procedure. The questionnaire was derived from the Group Health Association of America–9 (GHAA-9) survey (27), a previously validated instrument modified by the American Society for Gastrointestinal Endoscopy (ASGE) (mGHAA-9) to render it applicable to measurement of endoscopy patient satisfaction (28). The seven core items of the mGHAA-9 comprised our questionnaire, as illustrated in Table 1. We assigned a score of 1–5 to each item response, with 1 representing “poor” and 5 representing an “excellent” satisfaction rating. The maximal possible total satisfaction score was 35. Survey Methods STANDARD MAILED QUESTIONNAIRE. The questionnaire was mailed with a cover letter on a single occasion to those patients randomized to this mode within 1 wk of the endoscopic procedure. The cover letter assured the patient that response to the questionnaire was entirely voluntary and that all results would be strictly confidential, although there was no effort to maintain anonymity. An addressed stamped envelope for return mailing was enclosed with each questionnaire. The response rate and patient satisfaction score were recorded.
1018
Harewood et al.
TELEPHONE QUESTIONNAIRE. Each patient was queried at enrollment regarding the most convenient time for receiving telephone calls. Subjects randomized to this survey method were telephoned by a study coordinator at this time during the week after endoscopy. The interviewer began each interview by introducing herself to the patient; she then assured the patient that response to the questionnaire was entirely voluntary and that all results would be strictly confidential, although there was no effort to maintain anonymity. A patient’s telephone number was called on consecutive days at the same time of day until contact was made, up to a maximum of two attempts. Recognizing that we were calling these patients at times and days that they specifically suggested, we choose not to pursue patients with more than two telephone attempts. Mason et al. have demonstrated that extraordinary efforts to complete interviews with reluctant respondents may actually diminish the validity of the findings by introducing bias (29). The number of telephone calls made and response rate were recorded. WEB-BASED QUESTIONNAIRE. The Web address of the survey (http://www.mayosurvey.com/survey/run_survey.asp? surveyid ⫽ 18) and a unique password (patient clinic number) required for online survey submission were given to the patient on a card before endoscopy. The Web survey accepted one answer for each question and was submitted electronically by clicking on a “submit” button. Only completed questionnaires could be submitted online. Providing a unique password to each patient allowed us to accept only one completed survey per patient. Incompletely filled out mailed questionnaires were accounted for in the final analysis as being unanswered, using an intention-to-treat analysis. Patients in the mail or Web arms who did not respond within 4 wk to the original survey were considered to be nonresponders. These individuals were contacted with the same questionnaire by telephone after 4 wk to assess their satisfaction levels. The distribution of the satisfaction scores followed a normal distribution and were compared using Student’s t test for parametric data. Response rates were compared using Pearson’s 2 test for proportions or analysis of variance as appropriate. A multivariate logistic regression analysis was used to estimate the association between responder/ nonresponder status and mode of survey administration, with adjustment for the covariates age, sex, and type of endoscopic procedure (upper vs lower endoscopy). A level of statistical significance was defined as a p value of ⬍ 0.05 and the Bonferroni correction was applied where appropriate to correct for multiple comparisons. An estimated 50 patients were required in each arm of the study to detect a 20% difference in response rates to telephone, assuming an 88% telephone response rate, with 80% power at a two-sided ␣ level of 0.05. The estimated telephone response rate (88%) is based on the findings of Wilcox et al. (30). When at least 50 patients had been recruited in both arms of the non–Web user section of the
AJG – Vol. 98, No. 5, 2003
Table 2. Demographic Features of Non–Web Users and Web Users Non–Web Users (n ⫽ 106) Age, yr (mean, range) EGD (%) Colonoscopy (%) Sex (% male)
65.4 (20–88) 42 (40%) 64 (60%) 54 (51%)
Web Users (n ⫽ 159)
p
54.5 (31–86) ⬍0.0001 54 (34%) ns 105 (66%) ns 83 (52%) ns
study, we closed enrollment in that portion of the study and enrolled solely Web users.
RESULTS In total, 265 patients were approached to participate in this study and all provided informed verbal consent. Of the 106 patients enrolled as non–Web users, 52 were randomized to receive the survey by standard mail and 54 received it by telephone. Of the 159 patients enrolled as Web users, 55 were randomized to receive the survey by standard mail, 51 by telephone, and 53 were requested to access the survey online. Patient demographic data are given in Table 2. Although patient sex and procedure distributions did not differ significantly among the two groups, non–Web users were significantly older than Web users (p ⬍ 0.0001). There were no significant intragroup demographic differences when we compared the mail, telephone, and Web subgroups. In the non–Web users group, response rates for standard mail and telephone were 75% and 81%, respectively (p ⫽ ns) before follow-up telephone calls. In the Web users group, the response rate for the Web survey was 34%, inferior to both standard mail (67%, p ⫽ 0.0005) and telephone (78%, p ⬍ 0.0001), as illustrated in Figure 1. Of the 105 patients randomized to receive the survey by telephone, 90 patients were contacted after two attempts. Of these 90 subjects, 84 agreed to participate in the survey. None of the patients who completed the Web survey had attempted to complete the survey online a second time. The mean (⫾SEM) satisfaction scores in each group (maximal possible score, 35) were 31.9 ⫾ 0.8 for non–Web users and 31.3
Figure 1. Response rates in patient subgroups (non–Web users vs Web users). Dark gray bars indicate response by telephone; black bars, response by mail; light gray bars, response by use of the World Wide Web.
AJG – May, 2003
Web-Based Assessment of Patient Satisfaction
1019
Table 3. Response Rates and Satisfaction Scores of Patient Subgroups Non–Web Users
Response rate Mean satisfaction scores (SEM)
Web Users
Telephone (n ⫽ 54)
Mail (n ⫽ 52)
Telephone (n ⫽ 51)
Mail (n ⫽ 55)
Web (n ⫽ 53)
81% 32.6 ⫾ 0.6
75% 31.0 ⫾ 0.6
78%* 31.2 ⫾ 0.6
67%† 31.5 ⫾ 0.7
34% 31.0 ⫾ 0.9
* p ⬍ 0.0001 vs Web response rate. † p ⫽ 0.0005 vs Web response rate.
⫾ 0.6 for Web users. As illustrated in Table 3, there was no significant difference in satisfaction scores among the mail, telephone, and Web subgroups. We successfully contacted 82% of nonresponders (54 of 66 patients) by telephone 4 wk after their procedure. Their mean satisfaction scores were similar (nonresponders, 31.5 ⫾ 0.7 vs responders, 31.6 ⫾ 0.3; p ⫽ ns). On multivariate analysis, the only variable that was found to be predictive of a lack of response was randomization to the Web-based strategy (OR ⫽ 44.9, 25% CI ⫽ 7.9 –546.1, p ⫽ 0.0002) when controlling for age, sex, and type of procedure.
DISCUSSION The consequences of patient dissatisfaction with the health care that they receive have been well studied. Dissatisfied patients are more likely to be noncompliant (31–33), to allow their health insurance to lapse (34), to engage in litigation (35), and to “physician-shop” (36, 37). The use of both mail and telephone to distribute patient satisfaction surveys has also been extensively studied; however, use of the Internet is a newer approach that has not been evaluated in this context. In our study, unfortunately, the Web-based approach that we instituted to elicit responses to a patient satisfaction survey failed to demonstrate the potential that we had hoped. Specifically, it compared less favorably to standard mail and telephone in terms of response rates. Although we did not specifically elicit reasons for nonresponse from our patients, some of the following are likely to be responsible: non– user-friendly Web address, time required for completion of survey, and frequency of access to Web account. The cumbersome Web address that we requested patients to access was a legitimate barrier to the success of the Webbased approach. The length (59 characters) likely was not sufficiently user-friendly to minimize the time required for survey completion and to encourage responses. Furthermore, patients were also required to enter a seven-digit password that was provided at the time of enrollment. The combination of these two factors likely deterred patient responses. Although we arbitrarily defined Web access as at least once every 3 days, it is likely that the Web response rate would depend on frequency of Web access. Perhaps activation of an account once every 3 days may not be a strict enough definition. If the threshold were set higher (e.g., access on a daily basis), the Web response rate may have been higher.
What lessons can we learn from this study to improve upon questionnaire administration and data gathering with electronic media? We have already studied e-mail in this setting and demonstrated success with this tool (38). Our results in that study suggest that e-mail overcomes the deterrent of having to manually enter the Web page address (without mistakes). Furthermore, receipt of an e-mail message postprocedure serves as a reminder. Although e-mail’s application as a research tool seems to carry enormous potential, there are several concerns that exist. Many physicians are reluctant to use e-mail because of uncertainty surrounding the legal and ethical implications of communication with patients (18). There is confusion as to whether advice from physicians through e-mail assumes the status of official consultation. The development of future guidelines in this area should help to resolve some of this concern (39). Additionally, uncertainty exists regarding e-mail and privacy issues. The main utility of a Web-based survey is the ability to circumvent this issue by accepting only specific survey responses and not accepting any additional information. The ideal electronic tool will combine the user-friendliness of e-mail (no Web address required, reminder received by patient) and the legal safeguards of a Web page (allowing only the receipt of specific requested information). In this sense, an application whereby patients are automatically contacted through e-mail and directed to the survey Web page would be optimal. Unfortunately, in the interval between our initial e-mail– based study (38) and the initiation of this protocol, a moratorium was placed on e-mail solicitation of patients for health care–related purposes by the Mayo Clinic. This prevented us from either distributing the survey by e-mail or from e-mailing a reminder to patients to complete the survey. This arose from concerns generated by recent Health Insurance Portability and Accountability Act of 1996 regulations and the inability to ensure that the individual reading the e-mail was the patient. Until further guidance is provided by the Federal regulations, these restrictions will impede expansion of Web-based technology to assess satisfaction with health care. Should future guidelines sanction the use of e-mail communication between physicians and patients for health care matters, this medium would seem to hold much promise to enhance response rates, to shorten response times, and to lower costs for survey administration. Increasing numbers of physicians and health care institutions are maintaining Web sites (2, 14, 40 – 42)
1020
Harewood et al.
Use of the Internet for survey data distribution is constrained by its biased population of users in terms of age, income, sex, and ethnicity. This is supported by the younger average age of our Web user population compared with non–Web users. This limits the generalizability (external validity) of any survey findings obtained using this method. In a previous study, we demonstrated that only 45% of approached patients in the Mayo Clinic, Rochester, endoscopy suite had e-mail access (38). This low number may be reflective of the older age of the patients attending for endoscopy. Fridsma et al. observed an inverse relationship between a patient’s age and e-mail access ranging from 65% for those 20 –30 yr to 19% for those 70 – 80 yr (14). However, lack of access to the Internet is rapidly becoming less and less prevalent (14). The rate of adoption of the Internet is eclipsing that of all other technologies preceding it. In its first 4 yr of availability to the public, 50 million people logged on to the Internet worldwide. Currently, an estimated 40% of households have computers with more than 40 million people in the United States having Internet access (9). Today almost all libraries have Internet access, with the percentage of Internet-connected hospital libraries rising from 24% to 72% between 1993 and 1997 (43). In the past 4 yr the number of Internet hosts has increased 6-fold, and the number of domain names has increased almost 30-fold. Internet traffic doubles every 100 days; by the year 2005, 1 billion people worldwide are projected to be online according to the Department of Commerce. Although Web users still comprise a select population, our findings show that patient satisfaction scores did not differ significantly among the three modes of survey distribution. This suggests that Web users may be representative of the population in terms of endoscopy satisfaction. However, further studies would be required to validate this theory. Another aspect of this study that potentially limits its generalizability was its single center setting at the Mayo Clinic, Rochester. When one surveys a population of patients who have traveled from elsewhere to obtain health care at the Mayo Clinic, an element of selection bias has been introduced, as these patients may be more sensitive to their health care needs and therefore may be more motivated to participate in health care questionnaires than a representative sample of the population. The response rates that we found may therefore overestimate true response rates in the general population. Nevertheless, the pattern of superior response rates to telephone surveys compared with mail that we obtained is consistent with previous findings elsewhere (22, 30), and there is no reason to suspect that the satisfaction levels of these patients should differ significantly from other patient populations. Although we choose not to pursue patients with more than two telephone attempts (at times that were at their convenience), the persistence of the telephone interviewer can certainly influence response rate. There is no generally accepted standard number of telephone attempts, varying in the literature from one (44) or two (23) to six or more (45, 46). It seems that six attempts
AJG – Vol. 98, No. 5, 2003
are necessary to yield the maximal response, although the incremental effort involved for more than six attempts is unlikely to be rewarding (45) In summary, this study is the one of the first to address the role of the World Wide Web in context of patient satisfaction with endoscopic procedures. Although the selection bias of the study population somewhat limits the generalizability of the results, valuable insight is provided into the potential of this tool in surveying patients. Our findings provide feedback that will be helpful in devising an electronically based strategy to survey patients. As access to the Web becomes increasingly universal, the generalizability of responses to Web-based questionnaires will continue to improve. The primary challenge in implementing the technology in this setting is to maximize user-friendliness to improve response rates while incorporating safeguards that maintain the intent of recent Health Insurance Portability and Accountability Act regulations. Federal guidance to the research community is needed to allow expansion of this technology.
ACKNOWLEDGMENTS This work was supported by American Digestive Health Foundation/American Society for Gastrointestinal Endoscopy Clinical Outcomes and Effectiveness Award Funding. Reprint requests and correspondence: Gavin C. Harewood, M.D., M.Sc., Division of Gastroenterology and Hepatology, Mayo Clinic, 200 1st Street SW, Rochester, MN 55905. Received Sep. 30, 2002; accepted Jan. 22, 2003.
REFERENCES 1. Deering CM, Richardson J, Wares CG. Medical information: From Surgeon General to superhighway. Ann Trop Med Parasitol 1995;89:579 –91. 2. Neill RA, Mainous AG III, Clark JR, Hagen MD. The utility of electronic mail as a medium for patient-physician communication. Arch Fam Med 1994;3:268 –71. 3. Kim R, Kelly PJ. Applications of the World Wide Web to neurosurgical practice. Neurosurgery 1996;39:1169 –81, discussion 1181–2. 4. Rose S, Bruce J, Maffulli N. Accessing the Internet for patient information about orthopedics. JAMA 1998;280:1309. 5. Fernsler JI, Manchester LJ. Evaluation of a computer-based cancer support network. Cancer Pract 1997;5:46 –51. 6. Soetikno RM, Provenzale D, Lenert LA. Studying ulcerative colitis over the World Wide Web. Am J Gastroenterol 1997; 92:457–60. 7. Mold JW, Barton ED. OAFP starts practice-based resource/ research network. J Okla State Med Assoc 1996;89:433–4. 8. Alexander RB, Trissel D. Chronic prostatitis: Results of an Internet survey. Urology 1996;48:568 –74. 9. Wulkan ML, Smith SD, Whalen TV, Hardin WD Jr. Pediatric surgeons on the Internet: A multi-institutional experience. J Pediatr Surg 1997;32:612–4. 10. Murray PJ. Nurses’ computer-mediated communications on NURSENET. A case study. Comput Nurs 1996;14:227–34. 11. Worth ER, Patrick TB, Klimczak JC, Reid JC. Cost-effective
AJG – May, 2003
12. 13. 14.
15. 16. 17. 18. 19. 20. 21.
22. 23. 24. 25.
26. 27. 28.
clinical uses of wide-area networks: Electronic mail as telemedicine. Proc Annu Symp Comput Appl Med Care 1995: 814 – 8. Widman LE, Tong DA. Requests for medical advice from patients and families to health care providers who publish on the World Wide Web. Arch Intern Med 1997;157:209 –12. Branger PJ, Duisterhout JS. Communication in health care. Methods Inf Med 1995;34:244 –52. Fridsma DB, Ford P, Altman R. A survey of patient access to electronic mail: Attitudes, barriers, and opportunities. Proceedings—the Annual Symposium on Computer Applications in Medical Care 1994:15–9. Borowitz SM, Wyatt JC. The origin, content, and workload of e-mail consultations. JAMA 1998;280:1321–4. Eysenbach G, Diepgen TL. Responses to unsolicited patient e-mail requests for medical advice on the World Wide Web. JAMA 1998;280:1333–5. McDonald CJ, Overhage JM, Dexter PR, et al. Canopy computing: Using the Web in clinical practice. JAMA 1998;280: 1325–9. Spielberg AR. On call and online: Sociohistorical, legal, and ethical implications of e-mail for the patient-physician relationship. JAMA 1998;280:1353–9. Ferguson T. Digital doctoring– opportunities and challenges in electronic patient-physician communication. JAMA 1998;280: 1361–2. Neinstein L. Utilization of electronic communication (e-mail) with patients at university and college health centers. J Adolesc Health 2000;27:6 –11. Letterie GS, Morgenstern LL, Johnson L. The role of an electronic mail system in the educational strategies of a residency in obstetrics and gynecology. Obstet Gynecol 1994;84: 137–9. Sitzia J, Wood N. Response rate in patient satisfaction research: An analysis of 210 published studies. Int J Qual Health Care 1998;10:311–7. Harris LE, Weinberger M, Tierney WM. Assessing inner-city patients’ hospital experiences. A controlled trial of telephone interviews versus mailed surveys. Med Care 1997;35:70 –6. Skipper JK Jr, Ellison MD. Personal contact as a technique for increasing questionnaire returns from hospitalized patients after discharge. J Health Hum Behav 1966;7:211–4. Cohen MM, Duncan PG, Pope WD, et al. The Canadian four-centre study of anaesthetic outcomes: II. Can outcomes be used to assess the quality of anaesthesia care? Can J Anaesth 1992;39:430 –9. Melton LJ III. History of the Rochester Epidemiology Project. Mayo Clin Proc 1996;71:266 –74. Davies AR, Ware JE. GHAA’s consumer satisfaction survey and user’s manual. Washington, DC: Group Health Association of America, 1991. American Society for Gastrointestinal Endoscopy. ASGE Patient Satisfaction Survey (modified from GHAA-9), 1999.
Web-Based Assessment of Patient Satisfaction
1021
29. Mason R, Lesser V, Traugott MW. Impact of missing values from converted refusals on nonsampling error. International Conference on Survey Nonresponse Portland, OR, 1999. 30. Wilcox NS, Rossi JS, Prochaska JO. Rates and responses of distressed women to telephone survey screening and recruitment. Psychol Rep 1986;58:891–8. 31. Kallen DJ, Stephenson JJ. Perceived physician humaneness, patient attitude, and satisfaction with the pill as a contraceptive. J Health Soc Behav 1981;22:256 –67. 32. Francis V, Korsch BM, Morris MJ. Gaps in doctor-patient communication. Patients’ response to medical advice. N Engl J Med 1969;280:535–40. 33. Davis MS. Variation in patients’ compliance with doctors’ orders: Medical practice and doctor-patient interaction. Psychiatry Med 1971;2:31–54. 34. Davies AR, Ware JE, Jr, Brook RH, et al. Consumer acceptance of prepaid and fee-for-service medical care: Results from a randomized controlled trial. Health Serv Res 1986;21: 429 –52. 35. Vaccarino JM. Malpractice. The problem in perspective. JAMA 1977;238:861–3. 36. Kasteler J, Kane RL, Olsen DM, Thetford C. Issues underlying prevalence of “doctor-shopping” behavior. J Health Soc Behav 1976;17:329 –39. 37. Ware JE Jr, Davies AR. Behavioral consequences of consumer dissatisfaction with medical care. Eval Program Plann 1983; 6:291–7. 38. Harewood GC, Yacavone RF, Locke GR III, Wiersema MJ. Prospective comparison of endoscopy patient satisfaction surveys: E-mail versus standard mail versus telephone. Am J Gastroenterol 2001;96:3312–7. 39. Harris ED. Electronic mail. A physician extender? West J Med 1997;166:123–5. 40. Jones R, Pitt N. Health surveys in the workplace: Comparison of postal, email and World Wide Web methods. Occup Med (Lond) 1999;49:556 –8. 41. Raziano DB, Jayadevappa R, Valenzula D, et al. E-mail versus conventional postal mail survey of geriatric chiefs. Gerontologist 2001;41:799 –804. 42. Mold JW, Cacy JR, Barton ED. Patient-physician e-mail communication. J Okla State Med Assoc 1998;91:331–4. 43. Lindberg DA, Humphreys BL. Medicine and health on the Internet: The good, the bad, and the ugly. JAMA 1998;280: 1303–4. 44. Fowler FJ Jr. Comparing telephone and mail responses to the CAHPS survey instrument. Consumer Assessment of Health Plans Study. Med Care 1999;37(suppl 3):MS41–9. 45. Fowler FJ Jr, Gallagher PM, Stringfellow VL, et al. Using telephone interviews to reduce nonresponse bias to mail surveys of health plan members. Med Care 2002;40:190 –200. 46. Bell RA, Wilkes MS, Kravitz RL. Advertisement-induced prescription drug requests: Patients’ anticipated reactions to a physician who refuses. J Fam Pract 1999;48:446 –52.
Vol. 98, No. 5, 2003 ISSN 0002-9270/03/$30.00 doi:10.1016/S0002-9270(03)00104-7
Utility of Web-Based Assessment of Patient Satisfaction With Endoscopy Gavin C. Harewood, M.D., M.Sc., Maurits J. Wiersema, M.D., and Piet C. de Groen, M.D. Division of Gastroenterology and Hepatology, Mayo Clinic, Rochester, Minnesota
OBJECTIVES: The dramatic growth of the World Wide Web (Web) holds potential for use in survey distribution and submission. Its use has not previously been studied in the context of patient satisfaction with endoscopy procedures. In this study we compared standard mail, telephone, and Web-based modes of endoscopy satisfaction survey administration with respect to response rate and response content. METHODS: An endoscopy satisfaction questionnaire consisting of seven core items from the modified Group Health Association of America (GHAA-9) was distributed to patients after routine outpatient endoscopy. Patients were randomized to receive the questionnaire by standard mail, telephone, or Web (if applicable). Response rates and satisfaction scores in the groups were compared. The nonresponders to the standard mail and Web surveys were subsequently contacted by telephone to determine their level of satisfaction. RESULTS: Response rates to the telephone survey (81% among those designated as non–Web users and 78% among Web users) and standard mail (75% non–Web users, 67% Web users) were higher than response rate to the Web-based survey (34%) (p ⱕ 0.0005). There was no significant difference in satisfaction scores among the groups or between satisfaction of nonresponders and responders. CONCLUSIONS: The potential of our Web-based survey was limited by poor response rates. This arose from privacy protection precautions that complicated access to the Web site. Future Web-based strategies for surveying patients will need to be more user-friendly while maintaining the intent of the recent Health Insurance Portability and Accountability Act regulations. (Am J Gastroenterol 2003;98:1016 –1021. © 2003 by Am. Coll. of Gastroenterology)
INTRODUCTION The rapid permeation of new telecommunications technologies throughout society has seen the emergence of the World Wide Web (Web) as an increasingly pervasive means of communication. Throughout the 1990s, because of its relative simplicity and effectiveness, there has been explosive growth of the Internet and Web as tools for seeking and communicating health and medical information (1). Increasing numbers of physicians and health care institutions are
maintaining Web sites. Neill et al. found that 31% of patients in a family medicine practice had their primary physician’s electronic mail (e-mail) address, 37% corresponded with their practitioner, and 90% of this communication was related to a medical condition (2). In recent years, there has been a rapid growth in the literature addressing the role of the Internet in medicine. The applications of these modalities are evident in patient education (3, 4), patient support networks (5), patient follow-up (6 – 8), communication among providers (3, 4, 9 –11), and communication between providers and patients (2, 12–20). On an expanding scale, the medical education community has made efforts to introduce an electronic mail system for instruction (21). Although many medical information providers run Web pages, there have been few studies evaluating the use of the Internet as a mode of survey administration. Surveys conducted by traditional means, such as the mail and telephone, are commonly used to obtain information from large groups of people. However, these methods can be time-consuming and expensive and can yield low response rates (22, 23). Considerable delay can occur between the mailing of surveys and the receipt of responses. Receipt of surveys by respondents can be difficult to confirm. Respondents often find that filling out detailed questionnaires by hand is time consuming and that returning them may require several steps. Telephone calls may occur at inconvenient times and may require an immediate response, not allowing respondents time to carefully consider the questions. The use of a Web-based mode of survey administration holds potential to greatly facilitate patient satisfaction measurement. Although the use of Web-based surveys has been previously described (24), the application of this tool in gastroenterology—specifically, in endoscopy— has not been maximized. In clinical settings such as endoscopy, patient satisfaction is being increasingly used to monitor the quality of clinical care. For patients, satisfaction represents, at least in theory, an evaluation of the health care experience based on their own values, perceptions, and interactions with the health care environment (25). The primary goals of this study were: 1) to assess the feasibility of Web-based administration of endoscopy patient satisfaction surveys; 2) to compare response rates for this method with standard mail and telephone-based questionnaires; and 3) to compare
AJG – May, 2003
Web-Based Assessment of Patient Satisfaction
1017
Table 1. Modified GHAA-9 (Core Items): ASGE Patient Satisfaction Questionnaire In terms of your satisfaction, how would you rate each of the following: 1. How long you waited to get an appointment. Excellent Very good Good Fair 2. Length of time spent waiting at the office for the procedure. Excellent Very good Good Fair 3. The personal manner (courtesy, respect, sensitivity, friendliness) of the physician who performed your procedure. Excellent Very good Good Fair 4. The technical skills (thoroughness, carefulness, competence) of the physician who performed your procedure. Excellent Very good Good Fair 5. The personal manner (courtesy, respect, sensitivity, friendliness) of the nurses and the other support staff. Excellent Very good Good Fair 6. Adequacy of explanation of what was done for you—all your questions answered. Excellent Very good Good Fair 7. Overall rating of the visit Excellent Very good Good Fair
the satisfaction scores of immediate survey responders to the scores of nonresponders who were subsequently contacted.
MATERIALS AND METHODS Study Design and Methods The study was conducted in the outpatient endoscopy unit of Rochester Methodist Hospital, Mayo Medical Center (Rochester, MN). Approximately 15% of Mayo patients are residents of the city of Rochester or surrounding Olmsted County (26). Approximately 50% of Mayo patients live within 120 miles of Rochester. Thus, Mayo Medical Center provides primary, secondary, and tertiary level care. The Division of Gastroenterology and Hepatology offers open access esophagogastroduodenoscopy (EGD) and colonoscopy. Physicians from Mayo or outside referring physicians may order these procedures without prior consultation with or approval by a staff gastroenterologist. The study was reviewed and approved by the Mayo Clinic Institutional Review Board. All patients who were referred for routine outpatient EGD or colonoscopy and who had access to a telephone were potential study candidates. None of the investigators were involved in performing these procedures, and endoscopists were blinded to patient enrollment. Patients were recruited between June 2001 and December 2001. Exclusion criteria were patient refusal to participate or inability to give verbal consent. Importantly, patients were considered eligible for enrollment irrespective of their access to the Internet; lack of Web access resulted in randomization to one of two study arms, rather than one of three study arms. By stratifying patients into two study groups (“non–Web users” and “Web users”), this permitted us to compare the demographics, response rates, and satisfaction ratings of non–Web users and Web users and thereby to make assumptions about similarities and differences between the two populations. After enrollment, patients were classified as Web users or non–Web users on the basis of Web access. Access to the
Poor Poor Poor Poor Poor Poor Poor
Web was arbitrarily defined as possession of an account, either at home or at work, with use of this account at least once every 3 days. The questionnaire distribution strategy (mail, telephone, or e-mail) was written on cards and placed in separate sealed, opaque envelopes. Immediately before the endoscopy procedure, the study coordinator chose an envelope and thereby randomly assigned each patient to receive the questionnaire by standard mail, telephone, or e-mail. Patients were informed at the time of enrollment, before endoscopy, that they would receive a questionnaire by mail or telephone within 1 wk of their procedure or, alternatively, would be required to access the survey on a Web page if randomized to the Web arm of the study. To guard against the effect of retrograde amnesia induced by sedation, patients were also reminded of the study instructions after their procedure. The questionnaire was derived from the Group Health Association of America–9 (GHAA-9) survey (27), a previously validated instrument modified by the American Society for Gastrointestinal Endoscopy (ASGE) (mGHAA-9) to render it applicable to measurement of endoscopy patient satisfaction (28). The seven core items of the mGHAA-9 comprised our questionnaire, as illustrated in Table 1. We assigned a score of 1–5 to each item response, with 1 representing “poor” and 5 representing an “excellent” satisfaction rating. The maximal possible total satisfaction score was 35. Survey Methods STANDARD MAILED QUESTIONNAIRE. The questionnaire was mailed with a cover letter on a single occasion to those patients randomized to this mode within 1 wk of the endoscopic procedure. The cover letter assured the patient that response to the questionnaire was entirely voluntary and that all results would be strictly confidential, although there was no effort to maintain anonymity. An addressed stamped envelope for return mailing was enclosed with each questionnaire. The response rate and patient satisfaction score were recorded.
1018
Harewood et al.
TELEPHONE QUESTIONNAIRE. Each patient was queried at enrollment regarding the most convenient time for receiving telephone calls. Subjects randomized to this survey method were telephoned by a study coordinator at this time during the week after endoscopy. The interviewer began each interview by introducing herself to the patient; she then assured the patient that response to the questionnaire was entirely voluntary and that all results would be strictly confidential, although there was no effort to maintain anonymity. A patient’s telephone number was called on consecutive days at the same time of day until contact was made, up to a maximum of two attempts. Recognizing that we were calling these patients at times and days that they specifically suggested, we choose not to pursue patients with more than two telephone attempts. Mason et al. have demonstrated that extraordinary efforts to complete interviews with reluctant respondents may actually diminish the validity of the findings by introducing bias (29). The number of telephone calls made and response rate were recorded. WEB-BASED QUESTIONNAIRE. The Web address of the survey (http://www.mayosurvey.com/survey/run_survey.asp? surveyid ⫽ 18) and a unique password (patient clinic number) required for online survey submission were given to the patient on a card before endoscopy. The Web survey accepted one answer for each question and was submitted electronically by clicking on a “submit” button. Only completed questionnaires could be submitted online. Providing a unique password to each patient allowed us to accept only one completed survey per patient. Incompletely filled out mailed questionnaires were accounted for in the final analysis as being unanswered, using an intention-to-treat analysis. Patients in the mail or Web arms who did not respond within 4 wk to the original survey were considered to be nonresponders. These individuals were contacted with the same questionnaire by telephone after 4 wk to assess their satisfaction levels. The distribution of the satisfaction scores followed a normal distribution and were compared using Student’s t test for parametric data. Response rates were compared using Pearson’s 2 test for proportions or analysis of variance as appropriate. A multivariate logistic regression analysis was used to estimate the association between responder/ nonresponder status and mode of survey administration, with adjustment for the covariates age, sex, and type of endoscopic procedure (upper vs lower endoscopy). A level of statistical significance was defined as a p value of ⬍ 0.05 and the Bonferroni correction was applied where appropriate to correct for multiple comparisons. An estimated 50 patients were required in each arm of the study to detect a 20% difference in response rates to telephone, assuming an 88% telephone response rate, with 80% power at a two-sided ␣ level of 0.05. The estimated telephone response rate (88%) is based on the findings of Wilcox et al. (30). When at least 50 patients had been recruited in both arms of the non–Web user section of the
AJG – Vol. 98, No. 5, 2003
Table 2. Demographic Features of Non–Web Users and Web Users Non–Web Users (n ⫽ 106) Age, yr (mean, range) EGD (%) Colonoscopy (%) Sex (% male)
65.4 (20–88) 42 (40%) 64 (60%) 54 (51%)
Web Users (n ⫽ 159)
p
54.5 (31–86) ⬍0.0001 54 (34%) ns 105 (66%) ns 83 (52%) ns
study, we closed enrollment in that portion of the study and enrolled solely Web users.
RESULTS In total, 265 patients were approached to participate in this study and all provided informed verbal consent. Of the 106 patients enrolled as non–Web users, 52 were randomized to receive the survey by standard mail and 54 received it by telephone. Of the 159 patients enrolled as Web users, 55 were randomized to receive the survey by standard mail, 51 by telephone, and 53 were requested to access the survey online. Patient demographic data are given in Table 2. Although patient sex and procedure distributions did not differ significantly among the two groups, non–Web users were significantly older than Web users (p ⬍ 0.0001). There were no significant intragroup demographic differences when we compared the mail, telephone, and Web subgroups. In the non–Web users group, response rates for standard mail and telephone were 75% and 81%, respectively (p ⫽ ns) before follow-up telephone calls. In the Web users group, the response rate for the Web survey was 34%, inferior to both standard mail (67%, p ⫽ 0.0005) and telephone (78%, p ⬍ 0.0001), as illustrated in Figure 1. Of the 105 patients randomized to receive the survey by telephone, 90 patients were contacted after two attempts. Of these 90 subjects, 84 agreed to participate in the survey. None of the patients who completed the Web survey had attempted to complete the survey online a second time. The mean (⫾SEM) satisfaction scores in each group (maximal possible score, 35) were 31.9 ⫾ 0.8 for non–Web users and 31.3
Figure 1. Response rates in patient subgroups (non–Web users vs Web users). Dark gray bars indicate response by telephone; black bars, response by mail; light gray bars, response by use of the World Wide Web.
AJG – May, 2003
Web-Based Assessment of Patient Satisfaction
1019
Table 3. Response Rates and Satisfaction Scores of Patient Subgroups Non–Web Users
Response rate Mean satisfaction scores (SEM)
Web Users
Telephone (n ⫽ 54)
Mail (n ⫽ 52)
Telephone (n ⫽ 51)
Mail (n ⫽ 55)
Web (n ⫽ 53)
81% 32.6 ⫾ 0.6
75% 31.0 ⫾ 0.6
78%* 31.2 ⫾ 0.6
67%† 31.5 ⫾ 0.7
34% 31.0 ⫾ 0.9
* p ⬍ 0.0001 vs Web response rate. † p ⫽ 0.0005 vs Web response rate.
⫾ 0.6 for Web users. As illustrated in Table 3, there was no significant difference in satisfaction scores among the mail, telephone, and Web subgroups. We successfully contacted 82% of nonresponders (54 of 66 patients) by telephone 4 wk after their procedure. Their mean satisfaction scores were similar (nonresponders, 31.5 ⫾ 0.7 vs responders, 31.6 ⫾ 0.3; p ⫽ ns). On multivariate analysis, the only variable that was found to be predictive of a lack of response was randomization to the Web-based strategy (OR ⫽ 44.9, 25% CI ⫽ 7.9 –546.1, p ⫽ 0.0002) when controlling for age, sex, and type of procedure.
DISCUSSION The consequences of patient dissatisfaction with the health care that they receive have been well studied. Dissatisfied patients are more likely to be noncompliant (31–33), to allow their health insurance to lapse (34), to engage in litigation (35), and to “physician-shop” (36, 37). The use of both mail and telephone to distribute patient satisfaction surveys has also been extensively studied; however, use of the Internet is a newer approach that has not been evaluated in this context. In our study, unfortunately, the Web-based approach that we instituted to elicit responses to a patient satisfaction survey failed to demonstrate the potential that we had hoped. Specifically, it compared less favorably to standard mail and telephone in terms of response rates. Although we did not specifically elicit reasons for nonresponse from our patients, some of the following are likely to be responsible: non– user-friendly Web address, time required for completion of survey, and frequency of access to Web account. The cumbersome Web address that we requested patients to access was a legitimate barrier to the success of the Webbased approach. The length (59 characters) likely was not sufficiently user-friendly to minimize the time required for survey completion and to encourage responses. Furthermore, patients were also required to enter a seven-digit password that was provided at the time of enrollment. The combination of these two factors likely deterred patient responses. Although we arbitrarily defined Web access as at least once every 3 days, it is likely that the Web response rate would depend on frequency of Web access. Perhaps activation of an account once every 3 days may not be a strict enough definition. If the threshold were set higher (e.g., access on a daily basis), the Web response rate may have been higher.
What lessons can we learn from this study to improve upon questionnaire administration and data gathering with electronic media? We have already studied e-mail in this setting and demonstrated success with this tool (38). Our results in that study suggest that e-mail overcomes the deterrent of having to manually enter the Web page address (without mistakes). Furthermore, receipt of an e-mail message postprocedure serves as a reminder. Although e-mail’s application as a research tool seems to carry enormous potential, there are several concerns that exist. Many physicians are reluctant to use e-mail because of uncertainty surrounding the legal and ethical implications of communication with patients (18). There is confusion as to whether advice from physicians through e-mail assumes the status of official consultation. The development of future guidelines in this area should help to resolve some of this concern (39). Additionally, uncertainty exists regarding e-mail and privacy issues. The main utility of a Web-based survey is the ability to circumvent this issue by accepting only specific survey responses and not accepting any additional information. The ideal electronic tool will combine the user-friendliness of e-mail (no Web address required, reminder received by patient) and the legal safeguards of a Web page (allowing only the receipt of specific requested information). In this sense, an application whereby patients are automatically contacted through e-mail and directed to the survey Web page would be optimal. Unfortunately, in the interval between our initial e-mail– based study (38) and the initiation of this protocol, a moratorium was placed on e-mail solicitation of patients for health care–related purposes by the Mayo Clinic. This prevented us from either distributing the survey by e-mail or from e-mailing a reminder to patients to complete the survey. This arose from concerns generated by recent Health Insurance Portability and Accountability Act of 1996 regulations and the inability to ensure that the individual reading the e-mail was the patient. Until further guidance is provided by the Federal regulations, these restrictions will impede expansion of Web-based technology to assess satisfaction with health care. Should future guidelines sanction the use of e-mail communication between physicians and patients for health care matters, this medium would seem to hold much promise to enhance response rates, to shorten response times, and to lower costs for survey administration. Increasing numbers of physicians and health care institutions are maintaining Web sites (2, 14, 40 – 42)
1020
Harewood et al.
Use of the Internet for survey data distribution is constrained by its biased population of users in terms of age, income, sex, and ethnicity. This is supported by the younger average age of our Web user population compared with non–Web users. This limits the generalizability (external validity) of any survey findings obtained using this method. In a previous study, we demonstrated that only 45% of approached patients in the Mayo Clinic, Rochester, endoscopy suite had e-mail access (38). This low number may be reflective of the older age of the patients attending for endoscopy. Fridsma et al. observed an inverse relationship between a patient’s age and e-mail access ranging from 65% for those 20 –30 yr to 19% for those 70 – 80 yr (14). However, lack of access to the Internet is rapidly becoming less and less prevalent (14). The rate of adoption of the Internet is eclipsing that of all other technologies preceding it. In its first 4 yr of availability to the public, 50 million people logged on to the Internet worldwide. Currently, an estimated 40% of households have computers with more than 40 million people in the United States having Internet access (9). Today almost all libraries have Internet access, with the percentage of Internet-connected hospital libraries rising from 24% to 72% between 1993 and 1997 (43). In the past 4 yr the number of Internet hosts has increased 6-fold, and the number of domain names has increased almost 30-fold. Internet traffic doubles every 100 days; by the year 2005, 1 billion people worldwide are projected to be online according to the Department of Commerce. Although Web users still comprise a select population, our findings show that patient satisfaction scores did not differ significantly among the three modes of survey distribution. This suggests that Web users may be representative of the population in terms of endoscopy satisfaction. However, further studies would be required to validate this theory. Another aspect of this study that potentially limits its generalizability was its single center setting at the Mayo Clinic, Rochester. When one surveys a population of patients who have traveled from elsewhere to obtain health care at the Mayo Clinic, an element of selection bias has been introduced, as these patients may be more sensitive to their health care needs and therefore may be more motivated to participate in health care questionnaires than a representative sample of the population. The response rates that we found may therefore overestimate true response rates in the general population. Nevertheless, the pattern of superior response rates to telephone surveys compared with mail that we obtained is consistent with previous findings elsewhere (22, 30), and there is no reason to suspect that the satisfaction levels of these patients should differ significantly from other patient populations. Although we choose not to pursue patients with more than two telephone attempts (at times that were at their convenience), the persistence of the telephone interviewer can certainly influence response rate. There is no generally accepted standard number of telephone attempts, varying in the literature from one (44) or two (23) to six or more (45, 46). It seems that six attempts
AJG – Vol. 98, No. 5, 2003
are necessary to yield the maximal response, although the incremental effort involved for more than six attempts is unlikely to be rewarding (45) In summary, this study is the one of the first to address the role of the World Wide Web in context of patient satisfaction with endoscopic procedures. Although the selection bias of the study population somewhat limits the generalizability of the results, valuable insight is provided into the potential of this tool in surveying patients. Our findings provide feedback that will be helpful in devising an electronically based strategy to survey patients. As access to the Web becomes increasingly universal, the generalizability of responses to Web-based questionnaires will continue to improve. The primary challenge in implementing the technology in this setting is to maximize user-friendliness to improve response rates while incorporating safeguards that maintain the intent of recent Health Insurance Portability and Accountability Act regulations. Federal guidance to the research community is needed to allow expansion of this technology.
ACKNOWLEDGMENTS This work was supported by American Digestive Health Foundation/American Society for Gastrointestinal Endoscopy Clinical Outcomes and Effectiveness Award Funding. Reprint requests and correspondence: Gavin C. Harewood, M.D., M.Sc., Division of Gastroenterology and Hepatology, Mayo Clinic, 200 1st Street SW, Rochester, MN 55905. Received Sep. 30, 2002; accepted Jan. 22, 2003.
REFERENCES 1. Deering CM, Richardson J, Wares CG. Medical information: From Surgeon General to superhighway. Ann Trop Med Parasitol 1995;89:579 –91. 2. Neill RA, Mainous AG III, Clark JR, Hagen MD. The utility of electronic mail as a medium for patient-physician communication. Arch Fam Med 1994;3:268 –71. 3. Kim R, Kelly PJ. Applications of the World Wide Web to neurosurgical practice. Neurosurgery 1996;39:1169 –81, discussion 1181–2. 4. Rose S, Bruce J, Maffulli N. Accessing the Internet for patient information about orthopedics. JAMA 1998;280:1309. 5. Fernsler JI, Manchester LJ. Evaluation of a computer-based cancer support network. Cancer Pract 1997;5:46 –51. 6. Soetikno RM, Provenzale D, Lenert LA. Studying ulcerative colitis over the World Wide Web. Am J Gastroenterol 1997; 92:457–60. 7. Mold JW, Barton ED. OAFP starts practice-based resource/ research network. J Okla State Med Assoc 1996;89:433–4. 8. Alexander RB, Trissel D. Chronic prostatitis: Results of an Internet survey. Urology 1996;48:568 –74. 9. Wulkan ML, Smith SD, Whalen TV, Hardin WD Jr. Pediatric surgeons on the Internet: A multi-institutional experience. J Pediatr Surg 1997;32:612–4. 10. Murray PJ. Nurses’ computer-mediated communications on NURSENET. A case study. Comput Nurs 1996;14:227–34. 11. Worth ER, Patrick TB, Klimczak JC, Reid JC. Cost-effective
AJG – May, 2003
12. 13. 14.
15. 16. 17. 18. 19. 20. 21.
22. 23. 24. 25.
26. 27. 28.
clinical uses of wide-area networks: Electronic mail as telemedicine. Proc Annu Symp Comput Appl Med Care 1995: 814 – 8. Widman LE, Tong DA. Requests for medical advice from patients and families to health care providers who publish on the World Wide Web. Arch Intern Med 1997;157:209 –12. Branger PJ, Duisterhout JS. Communication in health care. Methods Inf Med 1995;34:244 –52. Fridsma DB, Ford P, Altman R. A survey of patient access to electronic mail: Attitudes, barriers, and opportunities. Proceedings—the Annual Symposium on Computer Applications in Medical Care 1994:15–9. Borowitz SM, Wyatt JC. The origin, content, and workload of e-mail consultations. JAMA 1998;280:1321–4. Eysenbach G, Diepgen TL. Responses to unsolicited patient e-mail requests for medical advice on the World Wide Web. JAMA 1998;280:1333–5. McDonald CJ, Overhage JM, Dexter PR, et al. Canopy computing: Using the Web in clinical practice. JAMA 1998;280: 1325–9. Spielberg AR. On call and online: Sociohistorical, legal, and ethical implications of e-mail for the patient-physician relationship. JAMA 1998;280:1353–9. Ferguson T. Digital doctoring– opportunities and challenges in electronic patient-physician communication. JAMA 1998;280: 1361–2. Neinstein L. Utilization of electronic communication (e-mail) with patients at university and college health centers. J Adolesc Health 2000;27:6 –11. Letterie GS, Morgenstern LL, Johnson L. The role of an electronic mail system in the educational strategies of a residency in obstetrics and gynecology. Obstet Gynecol 1994;84: 137–9. Sitzia J, Wood N. Response rate in patient satisfaction research: An analysis of 210 published studies. Int J Qual Health Care 1998;10:311–7. Harris LE, Weinberger M, Tierney WM. Assessing inner-city patients’ hospital experiences. A controlled trial of telephone interviews versus mailed surveys. Med Care 1997;35:70 –6. Skipper JK Jr, Ellison MD. Personal contact as a technique for increasing questionnaire returns from hospitalized patients after discharge. J Health Hum Behav 1966;7:211–4. Cohen MM, Duncan PG, Pope WD, et al. The Canadian four-centre study of anaesthetic outcomes: II. Can outcomes be used to assess the quality of anaesthesia care? Can J Anaesth 1992;39:430 –9. Melton LJ III. History of the Rochester Epidemiology Project. Mayo Clin Proc 1996;71:266 –74. Davies AR, Ware JE. GHAA’s consumer satisfaction survey and user’s manual. Washington, DC: Group Health Association of America, 1991. American Society for Gastrointestinal Endoscopy. ASGE Patient Satisfaction Survey (modified from GHAA-9), 1999.
Web-Based Assessment of Patient Satisfaction
1021
29. Mason R, Lesser V, Traugott MW. Impact of missing values from converted refusals on nonsampling error. International Conference on Survey Nonresponse Portland, OR, 1999. 30. Wilcox NS, Rossi JS, Prochaska JO. Rates and responses of distressed women to telephone survey screening and recruitment. Psychol Rep 1986;58:891–8. 31. Kallen DJ, Stephenson JJ. Perceived physician humaneness, patient attitude, and satisfaction with the pill as a contraceptive. J Health Soc Behav 1981;22:256 –67. 32. Francis V, Korsch BM, Morris MJ. Gaps in doctor-patient communication. Patients’ response to medical advice. N Engl J Med 1969;280:535–40. 33. Davis MS. Variation in patients’ compliance with doctors’ orders: Medical practice and doctor-patient interaction. Psychiatry Med 1971;2:31–54. 34. Davies AR, Ware JE, Jr, Brook RH, et al. Consumer acceptance of prepaid and fee-for-service medical care: Results from a randomized controlled trial. Health Serv Res 1986;21: 429 –52. 35. Vaccarino JM. Malpractice. The problem in perspective. JAMA 1977;238:861–3. 36. Kasteler J, Kane RL, Olsen DM, Thetford C. Issues underlying prevalence of “doctor-shopping” behavior. J Health Soc Behav 1976;17:329 –39. 37. Ware JE Jr, Davies AR. Behavioral consequences of consumer dissatisfaction with medical care. Eval Program Plann 1983; 6:291–7. 38. Harewood GC, Yacavone RF, Locke GR III, Wiersema MJ. Prospective comparison of endoscopy patient satisfaction surveys: E-mail versus standard mail versus telephone. Am J Gastroenterol 2001;96:3312–7. 39. Harris ED. Electronic mail. A physician extender? West J Med 1997;166:123–5. 40. Jones R, Pitt N. Health surveys in the workplace: Comparison of postal, email and World Wide Web methods. Occup Med (Lond) 1999;49:556 –8. 41. Raziano DB, Jayadevappa R, Valenzula D, et al. E-mail versus conventional postal mail survey of geriatric chiefs. Gerontologist 2001;41:799 –804. 42. Mold JW, Cacy JR, Barton ED. Patient-physician e-mail communication. J Okla State Med Assoc 1998;91:331–4. 43. Lindberg DA, Humphreys BL. Medicine and health on the Internet: The good, the bad, and the ugly. JAMA 1998;280: 1303–4. 44. Fowler FJ Jr. Comparing telephone and mail responses to the CAHPS survey instrument. Consumer Assessment of Health Plans Study. Med Care 1999;37(suppl 3):MS41–9. 45. Fowler FJ Jr, Gallagher PM, Stringfellow VL, et al. Using telephone interviews to reduce nonresponse bias to mail surveys of health plan members. Med Care 2002;40:190 –200. 46. Bell RA, Wilkes MS, Kravitz RL. Advertisement-induced prescription drug requests: Patients’ anticipated reactions to a physician who refuses. J Fam Pract 1999;48:446 –52.