- Email: [email protected]
Patient satisfaction and diagnostic accuracy with ultrasound by emergency physicians
Patient Satisfaction and Diagnostic Accuracy With Ultrasound by Emergency Physicians WILLIAM DURSTON, MD,* MICHAEL L. CARL, MD,* WAYNE GUERRA, MD-I In recent years, there has been considerable interest and controversy concerning the performance of ultrasound by emergency physicians (ED Sono), but patient satisfaction with ED Sono has not been well studied. The primary purpose of this investigation was to assess the level of patient satisfaction with ED Sono and to compare satisfaction with ED Sono with ultrasound by the Medical Imaging Department (MI Sono). A secondary objective was to assess the accuracy of ED Sono at our facility, During a 5-month period, which included the startup phase of a program for ED Sono, emergency physicians prospectively identified patients who were candidates for ultrasound as a part of their workup. Patients were contacted by telephone after their ED visit and asked to rate satisfaction on a 0 to 10 scale for various aspects of their care, including the ultrasound if one was done. The accuracy of ED Sono was determined by comparing ED ultrasound interpretations with surgical pathology, repeat imaging studies, or clinical follow-up. Two hundred forty patients were entered into the study, and 186 (78%) responded to the satisfaction survey. Satisfaction ratings were similarly high for ED Sono (mean, 8.9; 95% CI, 8.6 to 9.2) and for MI Sono (mean, 8.8; 95% CI, 8.2 to 9.4). Eighteen percent of ultrasounds performed by emergency physicians were indeterminate. Excluding indeterminate scans and scans for which confirmation was not possible, the accuracy of ED Sono was 99.1% (95% CI, 95.1% to >99.9%). We conclude that during the startup phase of our ED Sono program, patient satisfaction was high, and the error rate was very low. (Am J Emerg Med 1999;17:642-646. Copyright © 1999 by W.B. Saunders Company) Diagnostic ultrasound is of undisputed value in the evaluation of many conditions that bring patients to the Emergency Department (ED). Despite the great utility of ultrasonography in the ED, in many hospitals, including our own, there has been a reluctance on the part of Medical Imaging (Radiology) departments to routinely provide ultrasound studies on an immediate basis. 1 Over the past decade, there has been increasing interest among emergency physicians in performing ultrasonography themselves. A number of studies have shown that emergency physicians can accurately perform goal-directed ultrasound after relatively brief training periods. 24t Although there have been anecFrom the Division of Emergency Medicine, University of California, Davis, Sacramento, CA; the *Kaiser Foundation Hospital, South Sacramento, Sacramento, CA; and 1-Porter-Littleton Hospital, Denver, CO. Manuscript received September 26, 1998, returned November 5, 1998; revision received November 27, 1998, accepted November 27, 1998. Supported in part by a grant from the Kaiser Innovations Project, Kaiser Foundation Research Institute, Oakland, CA. Address reprint requests to Dr Durston, Emergency Department, Kaiser Foundation Hospital, 6600 Bruceville Rd, Sacramento, CA 95823. Key Words:Emergency department, ultrasound, satisfaction, quality, accuracy. Copyright © 1999 by W.B. Saunders Company 0735-6757/99/1707-0005510.00/0 642
dotal reports of favorable patient reactions, concerns also have been raised that patients may react negatively to having ultrasounds done by relatively inexperienced examiners. One small study has shown that patients presenting to the ED with threatened miscarriage were more satisfied when ultrasound by emergency physicians (ED Sono) was done than when no ultrasound was done during their ED visit. 12 No published study compares satisfaction with ED Sono versus satisfaction with ultrasound performed by the Medical Imaging Department (MI Sono). We obtained an ultrasound machine for use in our ED in March of 1996 and developed a protocol for training and credentialing emergency physicians to perform ED Sono. Over a period of several months, as our emergency physicians gained training and expertise in ED Sono, there was a transition from the initial point at which no physician in our department was performing ED Sono to the point at which all of our staff were relatively comfortable doing their own scans. We sought to take advantage of this transition period to compare satisfaction in patients who had ED Sono, patients who had MI Sono, and patients who were candidates for ultrasound but who did not have one performed during their ED visit. In this article, we present the results of our satisfaction survey, as well as data on our diagnostic accuracy during the startup phase of our ED ultrasound project.
MATERIALS AND METHODS The setting for the study was the ED of a large, staff model health maintenance organization (HMO). The annual ED census is approximately 30,000 patients. The hospital serves patients from urban, suburban, and semirural areas. Approximately 90% of ED patients are members of the health plan, but patients need not be health plan members to be seen in the ED. The hospital is a teaching institution affiliated with a major university medical center but is not a trauma center. During the period of this study, our ED physician staff consisted of 11 emergency physicians, all of whom were board certified in Emergency Medicine. Between March 1, 1996 and July 31, 1996, emergency physicians were asked to prospectively identify all patients who were candidates for an ultrasound examination as part of their ED workup, regardless of whether the patient actually had an ultrasound done during the ED visit, by stamping "SONO" on the ED copy of the patient's ED chart. On March 21, the ED acquired a General Electric RT3200 ultrasound machine (General Electric Corp., Milwaukee, WI) equipped with 3.5-MHz curved, 5.0-MHz endovaginal, and 7.5-MHz linear transducers and a Sony UP870MD black-and-white page printer (Sony Corp. of America, New York, NY). Four of the 11 emergency physicians on staff began doing ED Sonos at this time, under a protocol approved by the hospital Privileges and Credentials Committee. One of these 4 physicians had previously done a 1-month elective in ultrasound during his Emergency Medicine residency. The other 3 physicians had previously completed a 3-day, 24-hour course in ED ultrasound
DURSTON ET AL • SATISFACTION AND ACCURACY WITH ED ULTRASOUND
offered by Advanced Health Education of Houston, Texas. The ultrasound course included 12 hours of didactic training and 12 hours of practical experience on live models. From June 1 through June 3, 1996, the other 7 emergency physicians on the ED staff took the same course, and by June 4, all 11 emergency physicians were performing ED Sono. Under the hospital's ED ultrasound protocol, the first 25 ultrasounds performed by each emergency physician had to be repeated in Medical Imaging, though not necessarily at the time of the ED visit. For each ED Sono, the emergency physician completed a form documenting the indication for the scan and whether the scan was positive, negative, or indeterminate for the condition or pathology being sought. To advance from provisional to full ED ultrasound privileges, emergency physicians had to perform their first 25 ultrasounds with at least 90% accuracy, excluding indeterminate studies. When emergency physicians requested MI Sonos during the patient's ED visit, the scans were done in the ultrasound suite of the Medical Imaging Department, which is immediately adjacent to the ED, by full-time ultrasound technicians, using an Acuson XP-128 ultrasound machine (Acuson Corp., Mountain View, CA). During regular Medical Imaging Department hours, official readings of MI Sonos were done by radiologists on duty and phoned to the emergency physicians. At all other times, MI Sonos were interpreted by the ultrasound technician and validated by the ordering emergency physician, who viewed the relevant findings in real time with the technician. Radiologists reviewed the studies the next week day and dictated final reports, which were sent to the emergency physicians by interoffice mail. Before the beginning of the study, all ED personnel, radiologists, and ultrasound technicians were advised that a satisfaction study was being conducted, but they were not informed of the exact content of the questions being asked. Two specially trained ED receptionists were assigned to conduct telephone interviews of patients entered into the study, using a written script. Patients were asked to rate their level of satisfaction on a 0 to 10 scale for 12 different aspects of their ED care. The survey included questions regarding satisfaction with nonphysician staff, waiting time, and the physical condition of the ED. Embedded in the survey were 6 questions related to the ultrasound study. Patients also were asked how much they were charged for their ED visit. The interviewers attempted to make the first telephone contact as soon as possible after the patient's ED visit. In cases in which the patient himself could not answer the survey because of age, language barrier, or illness, an adult who accompanied the patient during the ED visit was asked to respond. If the patient or a proxy could not be contacted after 3 attempts, or if the patient declined to participate in the telephone interview, the patient was classified as a nonresponder. In addition to calling patients, interviewers abstracted relevant information from the ED record. The accuracy of ED Sono was confirmed by comparing ED ultrasound interpretations with surgical pathology, with repeat MI Sonos or other imaging studies, or with clinical follow-up. To be confirmed by clinical follow-up, cases had to meet strict, predefined criteria and had to be followed up at the health plan hospital or clinics for at least 6 months, or, in the case of pregnancy-related cases, to the conclusion of the patient's pregnancy. Statistical analysis of the data was done using the statistical packages included with Epilnfo version 6 (Centers for Disease Control, Atlanta, GA) and Microsoft Excel 97 (Microsoft Corp., Redmond, WA). P-values for comparisons of satisfaction survey scores were calculated using the Mann-Whitney or KruskalWallace tests for nonparametric data. Ages were compared using ANOVA. Other P-values were computed using chi-square. Ninetyfive percent confidence intervals for mean satisfaction scores were determined using the function, CONFIDENCE (.05, standard deviation, n) in Excel. This function calculates confidence intervals
643
as plus or minus the z-score times the sample standard deviation divided by the square root of n. Strictly speaking, 95% confidence intervals cannot be accurately determined for sample sizes less than 30 when the data are not normally distributed. 13 The 95% confidence intervals calculated by Excel for such subgroups in the study are referred to as "approximate" confidence intervals and are included to give the reader a general indication of the variability of the data.
RESULTS Two hundred forty patients were entered into the study. One hundred eighty-six patients (78%) responded to the satisfaction survey. Characteristics of responders and nonresponders are summarized in Table 1. The mean age and hospital admission rate were significantly higher for responders than for nonresponders. Otherwise, there were no statistically significant differences between responders and nonresponders in terms of gender, whether an emergency physician or ultrasound technician did the sonogram, or indications for the ultrasound examination. Indications for ultrasound were also compared for MI Sono versus ED Sono in patients who responded to the survey. The percentage of ultrasounds done to rule out deep venous thrombosis (DVT) was significantly higher for MI Sono as compared with ED Sono (14.8 v 3.6%, P = .01). The percentage of ultrasounds done to rule out intrauterine pregnancy was higher for ED Sono than for MI Sono (26.6 v 10.6%, P = .02). Otherwise, the indications for ED Sono and MI Sono were not significantly different. The average number of days from the ED visit to successful telephone contact with responders was 11.6 days, with a median interval of 6.5 and a range of less than 1 to 66 days. In 80% of cases, the responders were the patients themselves. Spouses or other caretakers responded in 16% of cases, and parents responded in 4%. TABLE1. Patient Characteristics
Characteristics
Responders Nonresponders (n = 186) (n = 54) P Value
Age 42.6 Females 142 (76%) Sono clone by ED physician only 126 (68%) Med. imaging only 34 (18%) Both 13 (6%) Neither 12 (5%) Gyn consultant 1 (1%) "Rule out" indications for sono AAA 16 (9%) Gallstones 59 (32%) Misc. abdominal* 23 (12%) IUP 44 (24%) Other pelvic pathology 15 (8%) DVT 11 (6%) Hydronephrosis 9 (5%) Other1 8 (4%) Admitted to hospital from ED 29 (16%)
35.7 39 (74%)
.03 .54 .91
35 (66%) 12 (23%) 3 (6%) 4 (6%) 0 .74 1 (2%) 17 (32%) 7 (13%) 14 (26%) 5 (9%) 5 (9%) 3 (6%) 1 (2%) 3 (6%)
.06
*Includes rule out ascites, hemoperitoneum, and appendicitis, as well as cases in which the physician sought to rule out more than 1 type of abdominal pathology. tlncludes rule out pleural/perieardial effusions and urinary retention.
644
AMERICAN JOURNAL OF EMERGENCY MEDICINE • Volume 17, Number 7 • November 1999
Table 2 lists the satisfaction questions that were asked relative to the ultrasound study, along with the mean satisfaction scores and 95% confidence intervals for each question. Questions are listed in the order in which they were asked on the telephone survey. The mean scores for satisfaction with ED Sono (8.9) and satisfaction with MI Sono (8.8) were not significantly different (P = .81). Breaking down satisfaction scores by type of ultrasound study, there was no significant difference between ED Sono and MI Sono for any single "rule out" indication. Thirty-eight percent of responders were not aware how much, if anything, they were charged for their ED visit. The average charge reported by the other 116 responders was $29.31, with a median charge of $5.00, and a range of zero to $1,800.00. Linear regression analysis showed a positive correlation between scores in each individual aspect of satisfaction, as assessed in questions 2 through 6, with overall satisfaction in question 1, but did not show any single satisfaction category to be significantly more important than the others in predicting overall satisfaction. The correlation coefficient between scores for question 5, satisfaction with ED Sono, and question 1, overall satisfaction, was r = .46 (95% confidence limits: .17 < r < .61). The correlation between question 6, satisfaction with MI Sono, and question 1 was r = .67 (95% confidence limits, .42 < r < .81). The amount that the patient was charged, as reported by the patient or other responder, showed no significant correlation with overall satisfaction (r = .11; 95% confidence intervals, - . 4 1 < r < .43). In Table 3, satisfaction scores are subdivided based on whether the patient had MI Sono, ED Sono during a physician's provisional credentialing period, ED Sono by a fully credentialed ED physician, both ED and MI Sono, or no ultrasound at all during the ED visit. The data in Table 3 demonstrate a trend toward higher scores for overall satisfaction, the caring attitude of the ED physician, the ED physician's skills and abilities, and the extent to which the patient's questions were answered when the patient had an ultrasound by a fully credentialled ED physician as compared with ultrasound by an ED physician during provisional credentialing. There was also a trend toward higher satisfaction scores in these questions for patients who had ED Sono as compared with MI Sono. Satisfaction scores for the ultrasound itself showed a trend toward highest scores when the ED Sono was done by a fully credentialed ED physician, and lowest scores for both ED Sono and MI Sono when both were done at the same visit. As indicated by the overlapping 95% confidence intervals in Table 3, for most comparisons of ultrasound categories, TABLE2. Mean Satisfaction Scores Question Number and Subject (No. of Respondents) 1. 2. 3. 4. 5. 6.
Overall satisfaction (186) Caring attitude of emergency physicians (186) Physicians' skills/abilities (186) Extent to which questions answered (184) Ultrasound by emergency physicians (131) Ultrasound by Medical Imaging (44)
95% Mean Confidence Score Intervals 7.8 9.1 9.1 8.5 8.9 8.8
7.4-8.2 8.9-9.3 8.8-9.4 8.2-8.8 8.6-9.2 8.2-9.4
TABLE3. Comparison of Satisfaction Ratings in Selected Questions by Ultrasound Category Question Number andSubject 1. Overall satisfaction 2. Caring attitude of ED physician 3. Physician's skills and abilities 4. Extent to which questions answered 5. Satisfaction with ED Sono 6. Satisfaction with MI Sono
No Sono MI Sono Both ED Prov. ED Full (n=12) (n=34) (n=13) (n=103) (n=23) 7.0 (2.3) 7.6 (0.9) 7.5 (1.3) 7.9 (0.5) 8.4 (0.9) 8.6 (1.8) 9.0 (0.6) 8.5 (1.0) 9.3 (0.3) 9.4 (0.4) 8.9 (1.8) 9.0 (0.7) 8.6 (1.3) 9.2 (0.3) 9.5 (0.4)
8.3 (1.9) 7.9 (1.0) 8.1 (1.6) 8.6 (0.4) 9.4 (0.4) N/A N/A
N/A
7.5 (1.9) 9.0 (0.4) 9.7 (0.3)
8.9 (0.8) 8.3 (1.6)
N/A
N/A
NOTE: Numbers in parentheses are +_95% confidence intervals. For subgroups with fewer than 30 patients, confidence intervals are approximations only. Mean satisfaction scores for selected questions are subdivided based on whether the patient had no ultrasound done during the ED visit (No Sono), ultrasound in medical imaging only (MI Sono), ultrasound both in medical imaging and by an ED physician at the same visit (Both), ultrasound only by an ED physician during his provisional credentialing period (ED Prov.), or ultrasound only by an ED physician who had attained full ultrasound privileges (ED Full).
observed differences were not statistically significant. For question 4, the extent to which the patient's questions were answered, the mean satisfaction score was significantly higher for patients who had ultrasound by a fully credentialed ED physician compared with patients who had ultrasound by an ED physician during provisional credentialing (9.4 v 8.6, P = .05) or compared with patients who had MI Sono (9.4 v 7.9, P = .04). For question 5, satisfaction with ED Sono, the mean satisfaction score was significantly higher for patients who had ED Sono by a fully credentialed ED physician compared with patients who had both ED Sono and MI Sono at the same visit (9.7 v 7.5, P = .04). All of the ED Sonos in the latter category were done by ED physicians during their provisional credentialing period. Of the 240 patients entered into the study, 178 had ED Sono. In 74 cases (42%), the ultrasound was interpreted as positive for the condition or pathology being sought. In 72 cases (40%), the ultrasound was negative. Thirty-two cases (18%) were considered indeterminate. In the 16 patients who had both ED Sono and MI Sono at the same visit, 4 ED Sonos (25%) were considered indeterminate. The frequencies of positive, negative, and indeterminate scans was not significantly different in responders versus nonresponders to the satisfaction survey (P = .82, calculated by chi-square). There was a trend toward fewer indeterminate scans as physicians gained experience. Of 145 scans performed by emergency physicians during their provisional credentialing period, 29 (20%) were indeterminate and 3 of the 33 (9%) scans performed after full credentialing were indeterminate. This difference did not reach statistical significance (P = . 14). All 146 cases in which emergency physicians interpreted scans as either positive or negative were reviewed for accuracy of the ultrasound interpretation. The results of this review are summarized in Table 4. Five cases (3%) were lost
DURSTON ET AL • SATISFACTION AND ACCURACY WITH ED ULTRASOUND
TABLE4. Results of Follow-Up to Confirm Accuracy of ED Sono
Confirmation Status
Positive Scans
Negative Scans
Totals
Confirmed by surgical pathology Confirmed by repeat imaging study Confirmed by clinical follow-up Unconfirmed with follow-up Lost to follow-up Error in ED sono Totals
15 26 21 7 4 1 74
1 23 26 21 1 0 72
16 49 47 28 5 1 146
to follow-up. In another 28 cases (19%), clinical follow-up was available through chart review, but the available data did not meet criteria for confirming whether the ED ultrasound interpretation was correct. In 112 cases (71%), the ED ultrasound interpretation was confirmed to be correct by surgical pathology, by a repeat ultrasound, by another imaging study, by clinical follow-up, or by a combination of these methods. Cases confirmed by more than 1 method are listed only once in Table 4, under the most stringent method, with surgical pathology being considered most stringent and clinical follow-up least stringent. Only 1 ED ultrasound interpretation was found to be in error, a false-positive study for gallstones. Excluding indeterminate scans and cases in which the patient was either lost to follow-up or follow-up could not definitely confirm whether the ED ultrasound interpretation was correct, the sensitivity of ED Sono was 100% (95% CI, 94.1% to 100%), the specificity 98.0% (95% CI, 89.6% to >99.9%), and the accuracy 99.1% (95% CI, 95.1% to >99.9%). Including indeterminate scans, the accuracy was 77.2% (95% CI, 69.5% to 83.8%). DISCUSSION
Our study found comparably high overall mean satisfaction scores for ED Sono (8.9) and for MI Sono (8.8) during the first 5 months of our ED ultrasound program. The 95% confidence intervals for these scores (8.6 to 9.2 for ED Sono, 8.2 to 9.4 for MI Sono) suggest that the real difference in satisfaction, if any, in our patient population is 1 point or less on a 10-point scale. When satisfaction scores were analyzed further, a trend was noted toward increasing satisfaction with ED Sono as ED physicians progressed from provisional to full credentialing. The lowest scores for satisfaction with ED Sono were observed in cases in which the ED physician ordered a repeat ultrasound in Medical Imaging at the same visit. This was also the only subgroup in which the mean satisfaction score for ED Sono was lower than for MI Sono. The relatively low scores for ED Sono in this subgroup are probably explained by the fact that the indeterminate rate for ED Sono in this subgroup was the highest (25%). Though none of the differences in mean ultrasound satisfaction scores between ED Sono and MI Sono in Table 3 reached statistical significance, the observed trend for mean scores for the various subgroups suggests that satisfaction was slightly lower with ED Sono than with MI Sono during the earliest stages of the credentialing process and slightly higher after full credentialing was attained, resulting in almost identical mean satisfaction scores for ED Sono and MI Sono when scores were averaged over the full 5-month study period.
645
Our study showed a trend toward higher mean scores for overall satisfaction with the ED visit, the caring attitude of the ED physician, and the physician's skills and abilities when the patient had ED Sono rather than MI Sono, and in particular when the ED Sono was done by a fully credentialed ED physician. The differences in favor of ED Sono in these categories were small, though, and not statistically significant. In the case of question 4, however, the extent to which the patient's questions were answered, there was a statistically significant difference in favor of ED Sono by a fully credentialed ED physician over MI Sono (9.4 v 7.9, P = .04). The explanation for this finding may lie in the difference in which ultrasounds are performed in Medical Imaging and in the ED. All ultrasounds done in Medical Imaging at our facility during regular hours are done by ultrasound technicians, who are not allowed to discuss the results with the patients during the study. When ultrasounds are done by ED physicians, however, the physicians are likely to discuss the results and their implications with the patients, whereas the studies are being done. Under these circumstances, it seems reasonable to expect that patients would tend to be more satisfied with the extent to which their questions are answered when ultrasounds are done by ED physicians, provided that the ultrasounds help establish definitive diagnoses. There was a trend toward lower satisfaction scores for patients who were candidates for ultrasound but who did not have a study performed during their ED visit, but the small number of patients in the "No Sono" category prevented any significant conclusions from being drawn. It has previously been shown in a study of patients presenting to the ED with first-trimester cramping and bleeding that patients who had ED Sono were significantly more satisfied with the number of tests performed and with overall ED care than patients who had no ultrasound done during their ED visit.12 We believe that the most likely reason that so few patients in the "No Sono" category were entered into the survey is that by the time we started the study, it had become the practice of most of the ED physicians at our facility to obtain an ultrasound during the ED visit in any patient who had indications for one, rather than ordering an ultrasound to be done at a later date. Because we practice in a setting in which most of our patients are prepaid health plan members, the charge to the patient for either ED Sono or MI Sono was largely a nonissue in our study. Our satisfaction results may not be extrapolatable to settings in which the patient incurs substantial charges for ultrasound studies. Conversely, the absence of significant financial ramifications as a result of ED or MI Sono in our study helps rule out any economic conflict of interest in ED physicians ordering or performing ultrasounds in the ED, and helps separate patient's satisfaction with ED charges from satisfaction with ED care. One limitation of our study is that ED physicians were the ones who identified patients to be entered into the satisfaction survey. In designing the study, we believed that this was the most practical way to prospectively identify all patients who were candidates for ultrasound during their ED visit. It is possible, though, that stamping "SONO" on the patient's chart could have served as a reminder to ED physicians to make a particular effort to satisfy the patient. The Medical Imaging Department was also aware that an ultrasound
646
AMERICAN JOURNAL OF EMERGENCY MEDICINE • Volume 17, Number 7 • November 1999
satisfaction survey was being conducted, and the ultrasound technicians may also have been making a special effort to satisfy patients. Given the current controversy concerning ED Sono, we believe that it is likely that even if the ED physicians had not been asked to identify patients entered into the study, they still would have made a special effort in cases in which they were doing ED Sono during the startup phase of the ED ultrasound program. Knowing that satisfaction with their care was being surveyed for patients who had MI Sono could possibly have led to increased effort to satisfy these patients as well. Our data on the accuracy of interpretation of ED ultrasounds support several prior studies showing that emergency physicians can reliably perform goal-directed sonography after relatively brief training. 2-11 Our accuracy of 99.1%, excluding indeterminate scans, is among the highest reported in the emergency medicine literature for a study of this size. Our overall indeterminate rate of 18%, though, is also the highest in the emergency medicine literature and deserves comment. The rate of indeterminate scans has not been specifically mentioned in most prior reports on ultrasound by emergency physicians. 2-4,6-9 Durham et al 5 reported an indeterminate rate of 12% in their study of ED Sono in 125 pregnant women with first-trimester cramping and bleeding. 5 Ingeman et all0 excluded 7 of 110 patients from analysis in their study of ED Sono for blunt abdominal trauma, 5 because of "poor image quality" and 2 because of "technical problems with the ultrasound unit. ''1° Shapiro and Nakamoto H reported an indeterminate rate of 12% for "nonspecialists" performing ultrasounds on 62 patients in a Japanese emergency department for a wide variety of indications. In setting up our ED ultrasound program, we encouraged our emergency physicians to record their ultrasound interpretations as indeterminate if they did not feel sure of their findings. We wrote our credentialing protocol in such a way that physicians would risk losing ultrasound privileges if they read scans as falsely positive or negative, but not if they read them as indeterminate. We anticipated a decline in the rate of indeterminate scans as our physicians gained experience, and this was borne out by the observed drop from 20% indeterminate scans during provisional credentialing to 9% after full credentialing. Besides inexperience, a number of factors contribute to indeterminate scans in the ED, and we believe they should lead emergency physicians to expect and accept an indeterminate rate in the range of 10% to 20%, regardless of their level of experience and expertise. The ultrasound machine we used for our study, like the machines used in most other EDs with ultrasound capability, was a compact portable unit, which sells for under $40,000. The ultrasound machines used in most Medical Imaging Departments, including our own, are large, relatively fixed units that cost many times that amount. Obviously, the resolution of these different machines is not the same. In addition, the lighting in the ED is often suboptimal for performing ultrasounds, patients do not usually present optimally "prepped," and emergency physicians do not have allotted appointment times for performing the scans. It is of note that in the study by Shapiro and Nakamoto, "trained ultrasonographers" had an indeterminate rate of 11%, which was not significantly different from the indeterminate rate of 12 % for "nonspecial-
ists," using the same ultrasound machine in the ED. The reported accuracy of ultrasound in this study, including indeterminate scans, was also not significantly different for "trained ultrasonographers" (67 % ) compared with"nonspecialists" (63%) using the same machine under the same conditions. Our accuracy of 77.2%, including indeterminate scans, compares favorably with these results. In conclusion, our study demonstrates that patient satisfaction with ED Sono was high, comparable to satisfaction with MI Sono, during the starmp phase of our ED ultrasound program, despite a relatively high indeterminate rate for ED Sono. There was a trend toward increasing satisfaction as ED physicians gained experience, and patient satisfaction with the extent to which their questions were answered was significantly greater with ED Sono than with MI Sono when studies were done by ED physicians who had completed at least 25 studies. Our study confirms previous studies demonstrating that ED physicians can perform ED Sono for a variety of indications with low error rates after relatively brief training periods. We believe that our study adds to the growing body of objective data demonstrating that the careful use of ultrasound by emergency physicians is in the best interest of patient care. The authors thank Michelle Wallner and Gwen Thompson for conducting the satisfaction interviews and extracting data from the ED records; Abigail Eaton, MPH, for help with data entry, statistical analysis, and review of the manuscript; and the emergency physicians in our ED for their participation in the study.
REFERENCES 1. Heller M, Crocco T, Patterson J, et al: Emergency ultrasound services as perceived by directors of radiology and emergency departments. Am J Emerg Med 1995;13:430-431 2. Mayron R, Gaudio FE, Plummer D, et al: Echocardiography performed by emergency physicians: Impact on diagnosis and therapy. Ann Emerg Med 1988;17:150-154 3. Jehle D, Davis E, Evans T, et al: A prospective study of ultrasonography in the E D by emergency physicians. Am J Emerg Med 1989;7:605-611 4. Schlager D, Lazzareschi G, Whitten D, et al: Emergency department sonography by emergency physicians. Am J Emerg Med 1994;12:185-189 5. Durham B, Lane B, Burbridge L, et at: Pelvic ultrasound performed by emergency physicians for the detection of ectopic pregnancy in complicated first-trimester pregnancies. Ann Emerg Med 1997;29:338-347 6. Shih CHY: Effect of emergency physician-performed pelvic sonography on length of stay in the emergency department. Ann Emerg Med 1997;29:348-352 7. Plummer D, Brunette D, Asinger R, et al: Emergency department echocardiography improves outcome in penetrating cardiac injury. Ann Emerg Med 1992;21:709-712 8. Mateer JR, Valley VT, Aiman EJ, et al: Outcome analysis of a protocol including bedside endovaginalsonography in patients at risk for ectopic pregnancy. Ann Emerg Med 1996;27:283-289 9. Ma OJ, Mateer JR: Trauma ultrasound versus chest radiography in the detection of hemothorax. Ann Emerg Med 1997;29:312314 10. Ingeman JE, Plewa MC, Okasinski RE, et al: Emergency physician use of ultrasonography in blunt abdominal trauma. Acad Emerg Med 1996;3:931-937 11. Shapiro RA, Nakamoto M: Ultrasonography in Japanese Emergency Departments. Am J Emerg Med 1990;8:443-445 12. Krubel R, Freedman D, Bursch B: Effects of transvaginal sonography on pregnant patients' perceptions of care in the ED (letter). Am J Emerg Med 1996;14:232-233 13. Koosis DJ: Statistics: A Self-Teaching Guide (ed 4). New York, NY, John Wiley & Sons, 1997, pp 82-96
dotal reports of favorable patient reactions, concerns also have been raised that patients may react negatively to having ultrasounds done by relatively inexperienced examiners. One small study has shown that patients presenting to the ED with threatened miscarriage were more satisfied when ultrasound by emergency physicians (ED Sono) was done than when no ultrasound was done during their ED visit. 12 No published study compares satisfaction with ED Sono versus satisfaction with ultrasound performed by the Medical Imaging Department (MI Sono). We obtained an ultrasound machine for use in our ED in March of 1996 and developed a protocol for training and credentialing emergency physicians to perform ED Sono. Over a period of several months, as our emergency physicians gained training and expertise in ED Sono, there was a transition from the initial point at which no physician in our department was performing ED Sono to the point at which all of our staff were relatively comfortable doing their own scans. We sought to take advantage of this transition period to compare satisfaction in patients who had ED Sono, patients who had MI Sono, and patients who were candidates for ultrasound but who did not have one performed during their ED visit. In this article, we present the results of our satisfaction survey, as well as data on our diagnostic accuracy during the startup phase of our ED ultrasound project.
MATERIALS AND METHODS The setting for the study was the ED of a large, staff model health maintenance organization (HMO). The annual ED census is approximately 30,000 patients. The hospital serves patients from urban, suburban, and semirural areas. Approximately 90% of ED patients are members of the health plan, but patients need not be health plan members to be seen in the ED. The hospital is a teaching institution affiliated with a major university medical center but is not a trauma center. During the period of this study, our ED physician staff consisted of 11 emergency physicians, all of whom were board certified in Emergency Medicine. Between March 1, 1996 and July 31, 1996, emergency physicians were asked to prospectively identify all patients who were candidates for an ultrasound examination as part of their ED workup, regardless of whether the patient actually had an ultrasound done during the ED visit, by stamping "SONO" on the ED copy of the patient's ED chart. On March 21, the ED acquired a General Electric RT3200 ultrasound machine (General Electric Corp., Milwaukee, WI) equipped with 3.5-MHz curved, 5.0-MHz endovaginal, and 7.5-MHz linear transducers and a Sony UP870MD black-and-white page printer (Sony Corp. of America, New York, NY). Four of the 11 emergency physicians on staff began doing ED Sonos at this time, under a protocol approved by the hospital Privileges and Credentials Committee. One of these 4 physicians had previously done a 1-month elective in ultrasound during his Emergency Medicine residency. The other 3 physicians had previously completed a 3-day, 24-hour course in ED ultrasound
DURSTON ET AL • SATISFACTION AND ACCURACY WITH ED ULTRASOUND
offered by Advanced Health Education of Houston, Texas. The ultrasound course included 12 hours of didactic training and 12 hours of practical experience on live models. From June 1 through June 3, 1996, the other 7 emergency physicians on the ED staff took the same course, and by June 4, all 11 emergency physicians were performing ED Sono. Under the hospital's ED ultrasound protocol, the first 25 ultrasounds performed by each emergency physician had to be repeated in Medical Imaging, though not necessarily at the time of the ED visit. For each ED Sono, the emergency physician completed a form documenting the indication for the scan and whether the scan was positive, negative, or indeterminate for the condition or pathology being sought. To advance from provisional to full ED ultrasound privileges, emergency physicians had to perform their first 25 ultrasounds with at least 90% accuracy, excluding indeterminate studies. When emergency physicians requested MI Sonos during the patient's ED visit, the scans were done in the ultrasound suite of the Medical Imaging Department, which is immediately adjacent to the ED, by full-time ultrasound technicians, using an Acuson XP-128 ultrasound machine (Acuson Corp., Mountain View, CA). During regular Medical Imaging Department hours, official readings of MI Sonos were done by radiologists on duty and phoned to the emergency physicians. At all other times, MI Sonos were interpreted by the ultrasound technician and validated by the ordering emergency physician, who viewed the relevant findings in real time with the technician. Radiologists reviewed the studies the next week day and dictated final reports, which were sent to the emergency physicians by interoffice mail. Before the beginning of the study, all ED personnel, radiologists, and ultrasound technicians were advised that a satisfaction study was being conducted, but they were not informed of the exact content of the questions being asked. Two specially trained ED receptionists were assigned to conduct telephone interviews of patients entered into the study, using a written script. Patients were asked to rate their level of satisfaction on a 0 to 10 scale for 12 different aspects of their ED care. The survey included questions regarding satisfaction with nonphysician staff, waiting time, and the physical condition of the ED. Embedded in the survey were 6 questions related to the ultrasound study. Patients also were asked how much they were charged for their ED visit. The interviewers attempted to make the first telephone contact as soon as possible after the patient's ED visit. In cases in which the patient himself could not answer the survey because of age, language barrier, or illness, an adult who accompanied the patient during the ED visit was asked to respond. If the patient or a proxy could not be contacted after 3 attempts, or if the patient declined to participate in the telephone interview, the patient was classified as a nonresponder. In addition to calling patients, interviewers abstracted relevant information from the ED record. The accuracy of ED Sono was confirmed by comparing ED ultrasound interpretations with surgical pathology, with repeat MI Sonos or other imaging studies, or with clinical follow-up. To be confirmed by clinical follow-up, cases had to meet strict, predefined criteria and had to be followed up at the health plan hospital or clinics for at least 6 months, or, in the case of pregnancy-related cases, to the conclusion of the patient's pregnancy. Statistical analysis of the data was done using the statistical packages included with Epilnfo version 6 (Centers for Disease Control, Atlanta, GA) and Microsoft Excel 97 (Microsoft Corp., Redmond, WA). P-values for comparisons of satisfaction survey scores were calculated using the Mann-Whitney or KruskalWallace tests for nonparametric data. Ages were compared using ANOVA. Other P-values were computed using chi-square. Ninetyfive percent confidence intervals for mean satisfaction scores were determined using the function, CONFIDENCE (.05, standard deviation, n) in Excel. This function calculates confidence intervals
643
as plus or minus the z-score times the sample standard deviation divided by the square root of n. Strictly speaking, 95% confidence intervals cannot be accurately determined for sample sizes less than 30 when the data are not normally distributed. 13 The 95% confidence intervals calculated by Excel for such subgroups in the study are referred to as "approximate" confidence intervals and are included to give the reader a general indication of the variability of the data.
RESULTS Two hundred forty patients were entered into the study. One hundred eighty-six patients (78%) responded to the satisfaction survey. Characteristics of responders and nonresponders are summarized in Table 1. The mean age and hospital admission rate were significantly higher for responders than for nonresponders. Otherwise, there were no statistically significant differences between responders and nonresponders in terms of gender, whether an emergency physician or ultrasound technician did the sonogram, or indications for the ultrasound examination. Indications for ultrasound were also compared for MI Sono versus ED Sono in patients who responded to the survey. The percentage of ultrasounds done to rule out deep venous thrombosis (DVT) was significantly higher for MI Sono as compared with ED Sono (14.8 v 3.6%, P = .01). The percentage of ultrasounds done to rule out intrauterine pregnancy was higher for ED Sono than for MI Sono (26.6 v 10.6%, P = .02). Otherwise, the indications for ED Sono and MI Sono were not significantly different. The average number of days from the ED visit to successful telephone contact with responders was 11.6 days, with a median interval of 6.5 and a range of less than 1 to 66 days. In 80% of cases, the responders were the patients themselves. Spouses or other caretakers responded in 16% of cases, and parents responded in 4%. TABLE1. Patient Characteristics
Characteristics
Responders Nonresponders (n = 186) (n = 54) P Value
Age 42.6 Females 142 (76%) Sono clone by ED physician only 126 (68%) Med. imaging only 34 (18%) Both 13 (6%) Neither 12 (5%) Gyn consultant 1 (1%) "Rule out" indications for sono AAA 16 (9%) Gallstones 59 (32%) Misc. abdominal* 23 (12%) IUP 44 (24%) Other pelvic pathology 15 (8%) DVT 11 (6%) Hydronephrosis 9 (5%) Other1 8 (4%) Admitted to hospital from ED 29 (16%)
35.7 39 (74%)
.03 .54 .91
35 (66%) 12 (23%) 3 (6%) 4 (6%) 0 .74 1 (2%) 17 (32%) 7 (13%) 14 (26%) 5 (9%) 5 (9%) 3 (6%) 1 (2%) 3 (6%)
.06
*Includes rule out ascites, hemoperitoneum, and appendicitis, as well as cases in which the physician sought to rule out more than 1 type of abdominal pathology. tlncludes rule out pleural/perieardial effusions and urinary retention.
644
AMERICAN JOURNAL OF EMERGENCY MEDICINE • Volume 17, Number 7 • November 1999
Table 2 lists the satisfaction questions that were asked relative to the ultrasound study, along with the mean satisfaction scores and 95% confidence intervals for each question. Questions are listed in the order in which they were asked on the telephone survey. The mean scores for satisfaction with ED Sono (8.9) and satisfaction with MI Sono (8.8) were not significantly different (P = .81). Breaking down satisfaction scores by type of ultrasound study, there was no significant difference between ED Sono and MI Sono for any single "rule out" indication. Thirty-eight percent of responders were not aware how much, if anything, they were charged for their ED visit. The average charge reported by the other 116 responders was $29.31, with a median charge of $5.00, and a range of zero to $1,800.00. Linear regression analysis showed a positive correlation between scores in each individual aspect of satisfaction, as assessed in questions 2 through 6, with overall satisfaction in question 1, but did not show any single satisfaction category to be significantly more important than the others in predicting overall satisfaction. The correlation coefficient between scores for question 5, satisfaction with ED Sono, and question 1, overall satisfaction, was r = .46 (95% confidence limits: .17 < r < .61). The correlation between question 6, satisfaction with MI Sono, and question 1 was r = .67 (95% confidence limits, .42 < r < .81). The amount that the patient was charged, as reported by the patient or other responder, showed no significant correlation with overall satisfaction (r = .11; 95% confidence intervals, - . 4 1 < r < .43). In Table 3, satisfaction scores are subdivided based on whether the patient had MI Sono, ED Sono during a physician's provisional credentialing period, ED Sono by a fully credentialed ED physician, both ED and MI Sono, or no ultrasound at all during the ED visit. The data in Table 3 demonstrate a trend toward higher scores for overall satisfaction, the caring attitude of the ED physician, the ED physician's skills and abilities, and the extent to which the patient's questions were answered when the patient had an ultrasound by a fully credentialled ED physician as compared with ultrasound by an ED physician during provisional credentialing. There was also a trend toward higher satisfaction scores in these questions for patients who had ED Sono as compared with MI Sono. Satisfaction scores for the ultrasound itself showed a trend toward highest scores when the ED Sono was done by a fully credentialed ED physician, and lowest scores for both ED Sono and MI Sono when both were done at the same visit. As indicated by the overlapping 95% confidence intervals in Table 3, for most comparisons of ultrasound categories, TABLE2. Mean Satisfaction Scores Question Number and Subject (No. of Respondents) 1. 2. 3. 4. 5. 6.
Overall satisfaction (186) Caring attitude of emergency physicians (186) Physicians' skills/abilities (186) Extent to which questions answered (184) Ultrasound by emergency physicians (131) Ultrasound by Medical Imaging (44)
95% Mean Confidence Score Intervals 7.8 9.1 9.1 8.5 8.9 8.8
7.4-8.2 8.9-9.3 8.8-9.4 8.2-8.8 8.6-9.2 8.2-9.4
TABLE3. Comparison of Satisfaction Ratings in Selected Questions by Ultrasound Category Question Number andSubject 1. Overall satisfaction 2. Caring attitude of ED physician 3. Physician's skills and abilities 4. Extent to which questions answered 5. Satisfaction with ED Sono 6. Satisfaction with MI Sono
No Sono MI Sono Both ED Prov. ED Full (n=12) (n=34) (n=13) (n=103) (n=23) 7.0 (2.3) 7.6 (0.9) 7.5 (1.3) 7.9 (0.5) 8.4 (0.9) 8.6 (1.8) 9.0 (0.6) 8.5 (1.0) 9.3 (0.3) 9.4 (0.4) 8.9 (1.8) 9.0 (0.7) 8.6 (1.3) 9.2 (0.3) 9.5 (0.4)
8.3 (1.9) 7.9 (1.0) 8.1 (1.6) 8.6 (0.4) 9.4 (0.4) N/A N/A
N/A
7.5 (1.9) 9.0 (0.4) 9.7 (0.3)
8.9 (0.8) 8.3 (1.6)
N/A
N/A
NOTE: Numbers in parentheses are +_95% confidence intervals. For subgroups with fewer than 30 patients, confidence intervals are approximations only. Mean satisfaction scores for selected questions are subdivided based on whether the patient had no ultrasound done during the ED visit (No Sono), ultrasound in medical imaging only (MI Sono), ultrasound both in medical imaging and by an ED physician at the same visit (Both), ultrasound only by an ED physician during his provisional credentialing period (ED Prov.), or ultrasound only by an ED physician who had attained full ultrasound privileges (ED Full).
observed differences were not statistically significant. For question 4, the extent to which the patient's questions were answered, the mean satisfaction score was significantly higher for patients who had ultrasound by a fully credentialed ED physician compared with patients who had ultrasound by an ED physician during provisional credentialing (9.4 v 8.6, P = .05) or compared with patients who had MI Sono (9.4 v 7.9, P = .04). For question 5, satisfaction with ED Sono, the mean satisfaction score was significantly higher for patients who had ED Sono by a fully credentialed ED physician compared with patients who had both ED Sono and MI Sono at the same visit (9.7 v 7.5, P = .04). All of the ED Sonos in the latter category were done by ED physicians during their provisional credentialing period. Of the 240 patients entered into the study, 178 had ED Sono. In 74 cases (42%), the ultrasound was interpreted as positive for the condition or pathology being sought. In 72 cases (40%), the ultrasound was negative. Thirty-two cases (18%) were considered indeterminate. In the 16 patients who had both ED Sono and MI Sono at the same visit, 4 ED Sonos (25%) were considered indeterminate. The frequencies of positive, negative, and indeterminate scans was not significantly different in responders versus nonresponders to the satisfaction survey (P = .82, calculated by chi-square). There was a trend toward fewer indeterminate scans as physicians gained experience. Of 145 scans performed by emergency physicians during their provisional credentialing period, 29 (20%) were indeterminate and 3 of the 33 (9%) scans performed after full credentialing were indeterminate. This difference did not reach statistical significance (P = . 14). All 146 cases in which emergency physicians interpreted scans as either positive or negative were reviewed for accuracy of the ultrasound interpretation. The results of this review are summarized in Table 4. Five cases (3%) were lost
DURSTON ET AL • SATISFACTION AND ACCURACY WITH ED ULTRASOUND
TABLE4. Results of Follow-Up to Confirm Accuracy of ED Sono
Confirmation Status
Positive Scans
Negative Scans
Totals
Confirmed by surgical pathology Confirmed by repeat imaging study Confirmed by clinical follow-up Unconfirmed with follow-up Lost to follow-up Error in ED sono Totals
15 26 21 7 4 1 74
1 23 26 21 1 0 72
16 49 47 28 5 1 146
to follow-up. In another 28 cases (19%), clinical follow-up was available through chart review, but the available data did not meet criteria for confirming whether the ED ultrasound interpretation was correct. In 112 cases (71%), the ED ultrasound interpretation was confirmed to be correct by surgical pathology, by a repeat ultrasound, by another imaging study, by clinical follow-up, or by a combination of these methods. Cases confirmed by more than 1 method are listed only once in Table 4, under the most stringent method, with surgical pathology being considered most stringent and clinical follow-up least stringent. Only 1 ED ultrasound interpretation was found to be in error, a false-positive study for gallstones. Excluding indeterminate scans and cases in which the patient was either lost to follow-up or follow-up could not definitely confirm whether the ED ultrasound interpretation was correct, the sensitivity of ED Sono was 100% (95% CI, 94.1% to 100%), the specificity 98.0% (95% CI, 89.6% to >99.9%), and the accuracy 99.1% (95% CI, 95.1% to >99.9%). Including indeterminate scans, the accuracy was 77.2% (95% CI, 69.5% to 83.8%). DISCUSSION
Our study found comparably high overall mean satisfaction scores for ED Sono (8.9) and for MI Sono (8.8) during the first 5 months of our ED ultrasound program. The 95% confidence intervals for these scores (8.6 to 9.2 for ED Sono, 8.2 to 9.4 for MI Sono) suggest that the real difference in satisfaction, if any, in our patient population is 1 point or less on a 10-point scale. When satisfaction scores were analyzed further, a trend was noted toward increasing satisfaction with ED Sono as ED physicians progressed from provisional to full credentialing. The lowest scores for satisfaction with ED Sono were observed in cases in which the ED physician ordered a repeat ultrasound in Medical Imaging at the same visit. This was also the only subgroup in which the mean satisfaction score for ED Sono was lower than for MI Sono. The relatively low scores for ED Sono in this subgroup are probably explained by the fact that the indeterminate rate for ED Sono in this subgroup was the highest (25%). Though none of the differences in mean ultrasound satisfaction scores between ED Sono and MI Sono in Table 3 reached statistical significance, the observed trend for mean scores for the various subgroups suggests that satisfaction was slightly lower with ED Sono than with MI Sono during the earliest stages of the credentialing process and slightly higher after full credentialing was attained, resulting in almost identical mean satisfaction scores for ED Sono and MI Sono when scores were averaged over the full 5-month study period.
645
Our study showed a trend toward higher mean scores for overall satisfaction with the ED visit, the caring attitude of the ED physician, and the physician's skills and abilities when the patient had ED Sono rather than MI Sono, and in particular when the ED Sono was done by a fully credentialed ED physician. The differences in favor of ED Sono in these categories were small, though, and not statistically significant. In the case of question 4, however, the extent to which the patient's questions were answered, there was a statistically significant difference in favor of ED Sono by a fully credentialed ED physician over MI Sono (9.4 v 7.9, P = .04). The explanation for this finding may lie in the difference in which ultrasounds are performed in Medical Imaging and in the ED. All ultrasounds done in Medical Imaging at our facility during regular hours are done by ultrasound technicians, who are not allowed to discuss the results with the patients during the study. When ultrasounds are done by ED physicians, however, the physicians are likely to discuss the results and their implications with the patients, whereas the studies are being done. Under these circumstances, it seems reasonable to expect that patients would tend to be more satisfied with the extent to which their questions are answered when ultrasounds are done by ED physicians, provided that the ultrasounds help establish definitive diagnoses. There was a trend toward lower satisfaction scores for patients who were candidates for ultrasound but who did not have a study performed during their ED visit, but the small number of patients in the "No Sono" category prevented any significant conclusions from being drawn. It has previously been shown in a study of patients presenting to the ED with first-trimester cramping and bleeding that patients who had ED Sono were significantly more satisfied with the number of tests performed and with overall ED care than patients who had no ultrasound done during their ED visit.12 We believe that the most likely reason that so few patients in the "No Sono" category were entered into the survey is that by the time we started the study, it had become the practice of most of the ED physicians at our facility to obtain an ultrasound during the ED visit in any patient who had indications for one, rather than ordering an ultrasound to be done at a later date. Because we practice in a setting in which most of our patients are prepaid health plan members, the charge to the patient for either ED Sono or MI Sono was largely a nonissue in our study. Our satisfaction results may not be extrapolatable to settings in which the patient incurs substantial charges for ultrasound studies. Conversely, the absence of significant financial ramifications as a result of ED or MI Sono in our study helps rule out any economic conflict of interest in ED physicians ordering or performing ultrasounds in the ED, and helps separate patient's satisfaction with ED charges from satisfaction with ED care. One limitation of our study is that ED physicians were the ones who identified patients to be entered into the satisfaction survey. In designing the study, we believed that this was the most practical way to prospectively identify all patients who were candidates for ultrasound during their ED visit. It is possible, though, that stamping "SONO" on the patient's chart could have served as a reminder to ED physicians to make a particular effort to satisfy the patient. The Medical Imaging Department was also aware that an ultrasound
646
AMERICAN JOURNAL OF EMERGENCY MEDICINE • Volume 17, Number 7 • November 1999
satisfaction survey was being conducted, and the ultrasound technicians may also have been making a special effort to satisfy patients. Given the current controversy concerning ED Sono, we believe that it is likely that even if the ED physicians had not been asked to identify patients entered into the study, they still would have made a special effort in cases in which they were doing ED Sono during the startup phase of the ED ultrasound program. Knowing that satisfaction with their care was being surveyed for patients who had MI Sono could possibly have led to increased effort to satisfy these patients as well. Our data on the accuracy of interpretation of ED ultrasounds support several prior studies showing that emergency physicians can reliably perform goal-directed sonography after relatively brief training. 2-11 Our accuracy of 99.1%, excluding indeterminate scans, is among the highest reported in the emergency medicine literature for a study of this size. Our overall indeterminate rate of 18%, though, is also the highest in the emergency medicine literature and deserves comment. The rate of indeterminate scans has not been specifically mentioned in most prior reports on ultrasound by emergency physicians. 2-4,6-9 Durham et al 5 reported an indeterminate rate of 12% in their study of ED Sono in 125 pregnant women with first-trimester cramping and bleeding. 5 Ingeman et all0 excluded 7 of 110 patients from analysis in their study of ED Sono for blunt abdominal trauma, 5 because of "poor image quality" and 2 because of "technical problems with the ultrasound unit. ''1° Shapiro and Nakamoto H reported an indeterminate rate of 12% for "nonspecialists" performing ultrasounds on 62 patients in a Japanese emergency department for a wide variety of indications. In setting up our ED ultrasound program, we encouraged our emergency physicians to record their ultrasound interpretations as indeterminate if they did not feel sure of their findings. We wrote our credentialing protocol in such a way that physicians would risk losing ultrasound privileges if they read scans as falsely positive or negative, but not if they read them as indeterminate. We anticipated a decline in the rate of indeterminate scans as our physicians gained experience, and this was borne out by the observed drop from 20% indeterminate scans during provisional credentialing to 9% after full credentialing. Besides inexperience, a number of factors contribute to indeterminate scans in the ED, and we believe they should lead emergency physicians to expect and accept an indeterminate rate in the range of 10% to 20%, regardless of their level of experience and expertise. The ultrasound machine we used for our study, like the machines used in most other EDs with ultrasound capability, was a compact portable unit, which sells for under $40,000. The ultrasound machines used in most Medical Imaging Departments, including our own, are large, relatively fixed units that cost many times that amount. Obviously, the resolution of these different machines is not the same. In addition, the lighting in the ED is often suboptimal for performing ultrasounds, patients do not usually present optimally "prepped," and emergency physicians do not have allotted appointment times for performing the scans. It is of note that in the study by Shapiro and Nakamoto, "trained ultrasonographers" had an indeterminate rate of 11%, which was not significantly different from the indeterminate rate of 12 % for "nonspecial-
ists," using the same ultrasound machine in the ED. The reported accuracy of ultrasound in this study, including indeterminate scans, was also not significantly different for "trained ultrasonographers" (67 % ) compared with"nonspecialists" (63%) using the same machine under the same conditions. Our accuracy of 77.2%, including indeterminate scans, compares favorably with these results. In conclusion, our study demonstrates that patient satisfaction with ED Sono was high, comparable to satisfaction with MI Sono, during the starmp phase of our ED ultrasound program, despite a relatively high indeterminate rate for ED Sono. There was a trend toward increasing satisfaction as ED physicians gained experience, and patient satisfaction with the extent to which their questions were answered was significantly greater with ED Sono than with MI Sono when studies were done by ED physicians who had completed at least 25 studies. Our study confirms previous studies demonstrating that ED physicians can perform ED Sono for a variety of indications with low error rates after relatively brief training periods. We believe that our study adds to the growing body of objective data demonstrating that the careful use of ultrasound by emergency physicians is in the best interest of patient care. The authors thank Michelle Wallner and Gwen Thompson for conducting the satisfaction interviews and extracting data from the ED records; Abigail Eaton, MPH, for help with data entry, statistical analysis, and review of the manuscript; and the emergency physicians in our ED for their participation in the study.
REFERENCES 1. Heller M, Crocco T, Patterson J, et al: Emergency ultrasound services as perceived by directors of radiology and emergency departments. Am J Emerg Med 1995;13:430-431 2. Mayron R, Gaudio FE, Plummer D, et al: Echocardiography performed by emergency physicians: Impact on diagnosis and therapy. Ann Emerg Med 1988;17:150-154 3. Jehle D, Davis E, Evans T, et al: A prospective study of ultrasonography in the E D by emergency physicians. Am J Emerg Med 1989;7:605-611 4. Schlager D, Lazzareschi G, Whitten D, et al: Emergency department sonography by emergency physicians. Am J Emerg Med 1994;12:185-189 5. Durham B, Lane B, Burbridge L, et at: Pelvic ultrasound performed by emergency physicians for the detection of ectopic pregnancy in complicated first-trimester pregnancies. Ann Emerg Med 1997;29:338-347 6. Shih CHY: Effect of emergency physician-performed pelvic sonography on length of stay in the emergency department. Ann Emerg Med 1997;29:348-352 7. Plummer D, Brunette D, Asinger R, et al: Emergency department echocardiography improves outcome in penetrating cardiac injury. Ann Emerg Med 1992;21:709-712 8. Mateer JR, Valley VT, Aiman EJ, et al: Outcome analysis of a protocol including bedside endovaginalsonography in patients at risk for ectopic pregnancy. Ann Emerg Med 1996;27:283-289 9. Ma OJ, Mateer JR: Trauma ultrasound versus chest radiography in the detection of hemothorax. Ann Emerg Med 1997;29:312314 10. Ingeman JE, Plewa MC, Okasinski RE, et al: Emergency physician use of ultrasonography in blunt abdominal trauma. Acad Emerg Med 1996;3:931-937 11. Shapiro RA, Nakamoto M: Ultrasonography in Japanese Emergency Departments. Am J Emerg Med 1990;8:443-445 12. Krubel R, Freedman D, Bursch B: Effects of transvaginal sonography on pregnant patients' perceptions of care in the ED (letter). Am J Emerg Med 1996;14:232-233 13. Koosis DJ: Statistics: A Self-Teaching Guide (ed 4). New York, NY, John Wiley & Sons, 1997, pp 82-96