The effects of an outpatient practice guideline at a teaching hospital: A prospective pilot study

The effects of an outpatient practice guideline at a teaching hospital: A prospective pilot study MICHAELG. STEWART,MD,MPH,WILLARDC. HARRILL,MD, and LAURIEA. OHLMS,MD, Houston, Texas Practice guidelines (PGs) are becoming increasingly important in modern medicine. To study the effects of a PG, we performed a pilot study at a large, urban, public teaching hospital according to a prospective, observational research design with both concurrent and historic controls. Specifically, we studied the effects of a multidisciplinary PG for pediatric outpatient tonsillectomy and adenoidectomy on the process of health-care delivery. Variables in the health-care process included patient compliance with clinic and surgery appointments, surgery time, operating room turnover, time in recovery room, unplanned admission rate, patient compliance with postoperative follow-up, provider compliance with guidelines, and hospital charges. Patients in the PG were found to have fewer preoperative laboratory tests, decreased duplication of services, and shorter operating room turnover times. Provider compliance with the PG varied by service and was intermittent at first but improved gradually. There was a trend toward improved compliance with postoperative foUow-up in patients in the PG. Provider opinions concerning the guideline were positive. This pilot study demonstrates several advantages and disadvantages of the use of PGs in the outpatient setting and in a teaching hospital. (Otolaryngol Head Neck Surg 1997;117:388-93,)

n the field of health services research, the delivery of health care has been divided into three stages: structure, process, and outcome. 1 Structure refers to the "inputs" into the system, such as patients and providers, process refers to what is done to the patient, and outcome refers to the consequences to the health and well-being of the patient. Because the delivery structure is typically not easy to change, health services researchers often study the process of health care, because the health-care process may affect clinical outcome. Practice guidelines (PGs), also known as critical pathways, clinical guidelines, or clinical pathways, are currently a popular method for standardizing the healthcare process. PGs have been defined as "systematically developed statements to assist practitioner decisions

From the Bobby R. AlfordDepartment of Otorhinolaryngologyand CommunicativeSciences,BaylorCollegeof Medicine. Presented at the Nineteenth Midwinter Research Meeting of the Associationfor Researchin Otolaryngology,St. Petersburg Beach, Fla., Feb. 4-8, 1996. Reprint requests: Michael G. Stewart, MD, MPH, Department of Otolaryngologyand CommunicativeSciences, Baylor College of Medicine, One BaylorPlaza (SM 1727), Houston,TX 77030. Copyright © 1997 by the AmericanAcademyof OtolaryngologyHead and Neck SurgeryFoundation, Inc. 0194~5998/97/$5.00 + 0 23/1/78279

388

about appropriate health care for specific clinical circumstances"2; it is assumed that PGs will either improve clinical outcome or at least maintain the same outcome while increasing efficiency and decreasing costs. The goals of this pilot study were to examine the implementation of a PG and its effects on the healthcare process. We chose an outpatient procedure (tonsillectomy and adenoidectomy) and an urban, public teaching hospital, because neither has been well studied to date on the use of PGs.

METHODS Development and implementation of the PG were performed at the Ben Taub General Hospital, Houston, Texas, a 550-bed, urban public hospital and a primary teaching hospital for Baylor College of Medicine. The patient population is primarily indigent, with approximately 25% of otolaryngology patients covered by Medicaid or Medicare. The PG itself was developed by a combination of methods. The 1995 Clinical Indicators Compendium, 3 which was developed by the Quality Improvement Committee of the American Academy of Otolaryngology-Head and Neck Surgery and thus constitutes "expert opinion," was used as a template on which the indications for surgery were based.

OtolaryngologyHead and Neck Surgery Volume 117 Number 4

STEWARTet el. 389

PRACTICE GUIDELINE: TONSILLECTOMY AND/OR ADENOIDECTOMY

ELiGiBILITY Age < 13 years Other concurrent procedures: PE tub~s INDICATIONS (CHECK ONE) 4 or more infections per year Hypertrophy causing upper airway symptom,,,(sleep disorder, severe dysphagia, cordiopulmonary complications) Pedtonsillar abscess Unilateral tonsillar hypertrophy Recurrent SOM (adenoids only) PATIENT CARD

PREOP

: PHYSICIAN Complete H&P document indids Obtain consent witness & translator complications Document referral source Pt has eligible gold card Order appt to preop screening Write pre-op note Order Hb/Hct ONLY

NURSE (CLINIC) Give preop screen appt (......./......./.......) Reinforce instruc's Witness consent Give directions to preop screening Provide eligibility info pm Check out Pt ANESTHESIOLOGIST Pt chart completed Pt appropriately prepared Other tests needed? Yes

NURSE (PREOP SCREENING) Medical chart complete Give wdtten instruc's for surgery Verbally reinforce written instruc's OR date (......J...,.../_.,...) Obtain consent for treatment Obtain consent for blood products Refused? Yes why. Old medical record avail

Other medical problem? Yes other tests

SURGERY

SURGEON Surgery technique electrocautery

DAY SURGERY CLERK Ardval time Did pt ardve on time? No why

NURSE (PRE.-OP AREA) Chart available Chart complete Surgery canceled? Yes why

sold dissection PETs placed concurrently lntraoperative complication?Yes

NURSE (O.R,) Time in room Actual surg start time _ _ _ _ Actual surg end _ _ _ Time out of r o o m , _ _

POST-OP

NURSE (PACU) Give follov~-up appt Reinforce post-op instructions

SURGEON Discharge home Admitted? Yes reeson Order follow-up appt: 1 month Order D/C meds antibiotic..._ ___ pain reed Order/give post-op instructions

Give D/C meds

PRACTICE GUIDELINE: TONSILLECTOMY AND/OR ADENOIDECTOMY

FOLLOW-

PHYSICIAN COMPLICATIONS (check all)

CLINIC APPT CLERK Date ....... I.......I .......

Hospital admission necessary Post-op bleeding Persistent nasal regurg Persistent voice change Delayed (>10 days) onset po intake Adverse medication reaction Recurrent infx PT STATUS Sleep pattern improved? Yes N/A Nasal breathing improved? Yes N/A Encourage =satisfaction surv~:y" Discharge from clinic Addt'l appt needed? Yes reason

Pt kept appt? Yes

u__~P

CLINIC NURSE Obtain (copy) satisfaction survey Give flu appt (if necessary) Reinforce instructions

Fig. 1. Flowsheet used with PG for outpatient tonsillectomy and adenoidectomy, PE, Pressure-equalization; SOM, secretory otitis media; H&P, history and physical examination; indic's, indications; instruc's, instructions; prn, as needed; OR, operating room; PETs, pressure-equalization ear tubes; appt, appointment; med, medication; D/C, discontinue; po, oral; infx, infections; N/A, not applicable; f/u, follow-up.

OtolaryngologyHead and Neck Surgery

390 STEWARTet al.

October 1997

Table 1. Comparison of entry variables for 26

patients who underwent outpatient tonsillectomy and adenoidectomy after a PG and 24 patients who underwent outpatient tonsillectomy and adenoidectomy not after a PG

Age (yr) Sex (proportion males) Indications Recurrent infections Hypertrophy Infections and hypertrophy Simultaneous positron emission tomograms

PG

Non-PG

8,9 0.50

7.5 0.50

0.15 0.27 0.27

0.25 0.25 0.21

0.31

0.29

pValue

0.24 1.0 0.38

"Consensus building" was used to construct the elements of the PG and the flowsheet itself. Multidisciplinary meetings were held with all providers along the path of outpatient care (e.g., otolaryngology residents and faculty, the outpatient clinic nursing staff, anesthesiologists, and operating room nursing staff), and the local standard of best care was defined and organized on a PG "flowsheet." Providers designing the PG identified several potential points of inefficiency within the system, and the PG was designed to increase efficiency at those points in the process. The PG was "evidence based" whenever possible; where scientific evidence existed to support a management decision, it was included. Specific examples include the use of no routine preoperative blood coagulation studies, 4,5 the use of outpatient surgery, 6-10 and the routine use of postoperative antibiotics. 11 The PG flowsheet was not a permanent part of the medical record but was attached to the outpatient chart (Fig. 1). This was intentional because the flowsheet was not designed as a set of orders that must be followed but rather as a checklist or "guide" to direct care. Once the PG flowsheet was developed, the otolaryngology residents, staff, and supervisors of other patientcare areas were oriented to its use, and these supervisors then oriented and instructed their employees. Otolaryngology residents acted as primary caretakers; the decision whether to enter a patient into the PG was voluntary on the part of the resident. After the pathway had been in place for 6 months, flowsheets were collected and medical charts were reviewed. In addition, the operative schedule was reviewed to identify all patients who had undergone tonsillectomy and adnoidectomy who were not in the PG; those charts were also reviewed. Process variables were

retrieved from the flowsheet, the medical record, operative services records, and the computerized appointment system for outpatient clinics. In addition, a convenience sample of charts and records of patients who had undergone tonsillectomy and adnoidectomy in the 12 months before initiation of the PG were also reviewed and process variables retrieved. In total, 26 patients who used the PG and 24 patients who did not follow the PG were studied. Six months after initiation of the pathway, an anonymous survey on provider attitudes and beliefs was distributed to a convenience sample from all groups of providers who participated in the PG. Statistical analysis was performed in the following manner: Dichotomous categoric variables were compared with the Z 2 test. Ordinal and continuous variables were compared with the Mann-Whitney U test; t tests were'not used because the population was fairly small, and we were unwilling to assume an underlying normal distribution. RESULTS

During the first 5 months that the PG was available, 44 eligible patients underwent tonsillectomy and adnoidectomy, and 26 patients (58%) were enrolled in the PG. The compliance with the PG improved as time progressed. In the first 3 months only 50% (14 of 28) of eligible patients were enrolled, but in the last 2 months 71% (12 of 17) were enrolled. Analysis of entry variables (age, sex, and indications for surgery) revealed no difference in the PG and non-PG groups (Table 1), making selection bias for PG entry unlikely. Process variables for the two groups are compared in Table 2. The PG had a significant impact on the number of preoperative laboratory studies ordered. Patients off the PG typically underwent several laboratory tests, including urinalysis and coagulation studies, and often had duplication of tests. For example, tests ordered by the otolaryngology service were often not on the chart at the time of the preoperative screening clinic visit and were then repeated by the anesthesia service; in some cases patients underwent six laboratory tests. The PG did not affect compliance with appointment keeping for surgery or the efficiency of surgery scheduling, although compliance was overall very high and surgery scheduling was completed within 1 week in both groups. Operating room time and postanesthesia care unit (PACU) time were not affected by the PG, although turnover time between cases was shorter in patients in the PG. There was also a trend toward improved compliance with postoperative follow-up in patients in the PG. We compared hospital charges for patients in and not

OtolaryngologyHead and Neck Surgery

STEWARTet ai, 391

Volume 117 Number 4

Table 2. C o m p a r i s o n of process variables for 26

Table 3. C o m p l i a n c e with c o m p l e t i o n of t h e

patients who underwent outpatient tonsillectomy a n d a d e n o i d e c t o m y using a PG a n d 24 p a t i e n t s who underwent outpatient tonsillectomy and a d e n o i d e c t o m y not using a PG

a p p r o p r i a t e portion of t h e PG f l o w s h e e t in 26 patients

No. of preoperative laboratories On time for surgery (proportion) Days until surgery scheduled Surgical technique (proportion using cautery) Operating room time (min) Operating room turnover (min) PACU time (min) Admitted (proportion) Major complication (proportion) Kept follow-up clinic appointment (proportion) Total hospital charges ($)

PG

Non-PG

p Value

1.5

2,2

0.0002

0.96

0,90

0.49

5.12

5.13

0.69

0,77

0.C7

0.42

43.4 19,0

46.~ 31.4

0.69 0,002

276 0.15 0.08

274 0.25 0.04

0.77 0.40 0.84

0.72

0.4.8

0.09

3467

3539

0.66

B

Preoperative physician Outpatient clinic nurse Anesthesia physician Preoperative screening clinic nurse Day Surgery Unit clerk Preoperative holding area nurse Operating surgeon Operating room nurse Surgeon in PACU PACU nurse Follow-up physician Follow-up clinic nurse

I00 5O 46 58 89 92 85 92 89 69 31 23

Table 4. Results of the " p r o v i d e r beliefs" survey

g i v e n 6 m o n t h s after initiation of t h e PG (n : 29) i

in the PG, and we compared charges for patients before June 1995 and after June 1995 (date of initiation of the PG), regardless of whether the patient was in the PG, and neither difference was statistically significant. Further, an analysis of charges by quarter did not reveal a significant trend in charges after initiation of the PG. However, power analysis for the difference in total charges revealed that, to detect a difference of $200 from a mean of $3500 (SD = $700; type I error = 0.05; power = 0.80), 193 patients per group were required. Therefore a type II error was possible, and our pilot study was underpowered to detect small differences in charges. Compliance with completion of the appropriate portion of the PG flowsheet for each of the providers is listed in Table 3. Because the physician initiating the preoperative process is a necessary step to begin the PG, compliance was, by definition, 100% Compliance was related to the number of providers at each point in the pathway. When there were relatively few providers completing the forms on a regular basis, such as the operating room nurse and the clerk in the Day Surgery Unit, compliance was high. In contrast, compliance was low when there was high provider turnover from month to month, such as the anesthesiology service and the outpatient clinic. Further, compliance with completion of the PG flowsheet during the patient's follow-up visit was poor: only eight (31%) of 26 flowsheets were completed. This was mostly due to lack of availability of

%

PG flowsheet

Yes

Unsure

No

21

4

4

29

0

0

23

4

2

25

2

2

i

Have you cared for an otolaryngologic patient on the PG? De you believe PGs in general are helpful in patient care? Do you believe the otolaryngologic PG in particular is helpful in patient care? Are you interested in learning more about PGs in general?

Easier No change Harder

If you had several patients with several different PGs, would it make your job:

11

t0

8

the flowsheet itself, because the flowsheet was separated from the medical record in the PACU. The results of the provider beliefs survey are detailed in Table 4. Providers had overall positive opinions about PGs in general, and many were interested in learning more about PGs. Interestingly, however, opinions were mixed on whether use of multiple PGs would be helpful. DISCUSSION

PGs, also known as critical pathways or clinical pathways, have actually been in existence in the medical literature for many years.12 Although many of the first PGs were developed for high-risk, high-cost illnesses such as cardiovascular disease and stroke, guidelines are currently under development and testing in

392 STEWARTet al.

virtually every medical specialty. The goals of PGs may differ, depending on the interested party; patients, insurers, employers and other payers, health care administrators, physicians, and attorneys have different perspectives. 12 Although there are several potential benefits that could be achieved with PGs, there is controversy over whether simply developing and distributing a guideline actually improves health-care outcomes. 12-16 Similarly, there is controversy on the strength of the recommendations made in a PG and the degree to which guidelines should be applicable to all patients. 17 Eddy 18 addressed the issue of the strength behind policy recommendations by defining practice standards as rigid practices to be applied in every case, whereas PGs should be followed in most cases. In this study the terminology PG was chosen because we wanted to offer clinicians an option without forcing compliance. We also purposely chose a public teaching hospital as the implementation site because of several perceived theoretic pitfalls against development and use of PGs: noncompliance of the patient population, turnover among hospital housestaff and ancillary staff, "large institution inertia," and frequent long waiting lines f o r hospital services. We reasoned that if a PG could effect positive changes despite these inherent difficulties, the PG method definitely has merit. The use of a teaching hospital did introduce some shortcomings into the study, however. Specifically, the use of junior residents in training as primary surgeons and caretakers limited the degree to which we could control all sources of variation. Certainly, mandating that only the most experienced residents perform tonsillectomies (or, for instance, that only electrocautery dissection be used) might have decreased the median operating room time, but this was in conflict with the hospital's primary teaching mission. Therefore our study was not intended to minimize costs or maximize efficiency but rather to measure the effect of a voluntary PG on the health-care process given the constraints of the system. We identified several potential points of inefficiency within the system, and the PG was designed to maximize efficiency at those points in the process. Specific examples of process steps identified as sources of significant variance were determination of hospital eligibility, prompt completion of necessary forms, preoperative laboratory testing, reinforcement of patient teaching to decrease the no-show rate, and timing of the postoperative follow-up visit; all these were addressed in the PG. Several methods are described for developing PGs, and most authors report that a combination of methods is often useful. 19,20The evidence-based approach is the

OtolaryngologyHead and Neck Surgery October 1997

most appealing from a scientific standpoint. Before any intervention can be placed into the guideline, there must be solid experimental evidence (preferably by randomized clinical trial) that the intervention is beneficial. Although this approach is ideal, clinical trials are, unfortunately, not available to answer every potential clinical question, and often results of different trials are contradictory. An alternative approach is to use expert opinion in the PG, such as that obtained from a consensus conference or a position statement from an academic governing body. The consensus-development approach refers to collaboration between practitioners for the development of algorithms that all agree to implement. The algorithm agreed on may be based on scientific evidence, expert opinion, prevailing local standard of practice, or a combination of those. The use of the consensus-development approach was crucial in our PG. Input, cooperation, and enthusiasm from all services involved greatly facilitated the implementation of the PG, and involving representatives from other services in PG development is an important step. Most published reports on the effects of PGs have found improvements in costs of care, process of care, or outcome after PGs, 2,12,20 although the "size of the improvements in performance varied considerably. ''2 Similarly, in this study we identified improvements in the process of care, although they were not always as predicted or hoped. We did identify a significant problem with ordering unnecessary preoperative laboratory tests (and different physicians ordering two sets of the same tests); this was noted by the preoperative clinic nursing staff during completion of the PG flowsheet. Similarly, we noted a trend toward improved postoperative follow-up in patients in PGs, which we attribute to the use of a standard order and a double check by the PACU nurse that the appointment was given. We had hoped that compliance with preoperative screening clinic and surgery appointments would be improved with the PG; however, compliance was fairly high in both PG and non-PG groups, and we found no improvement with the PG. The operating room turnover was faster in patients in PGs, but the reasons for this were unclear. Although use of the PG may have directly or indirectly improved the efficiency of the health-care process, perhaps only random factors (such as overall number of cases posted for the day, the availability of more than one anesthesiology resident per room, or the quality and experience of operating room technicians) caused the difference in turnover times between the two groups. Certainly, delays caused by paperwork and charting problems should have been minimized by use of the PG.

OtolaryngologyHead and Neck Surgery Volume 117 Number 4

Hospital charges were not affected by the use of the PG. Although cost data are preferable over charges, costs are much more difficult to retrieve at our county hospital. Comparison with charges for the historic controls also showed no difference before and after initiation of the PGs. However, because the primary components of the hospital charge for day-surgery patients are operating room time and PACU time, it is not surprising that overall charges were not different. We found that a significant problem was keeping the flowsheet with the chart when the patient returned f o r follow-up. The PG flowsheet was not part of the permanent medical record because, at our hospital, patients usually have separate hospital and clinic chart volumes, and the entire chart often is not available at the time of clinic appointments. When the patient was discharged from the PACU, the flowsheet was therefore removed from the chart and placed in the clinic. Despite this, however, man 3, times at the follow-up visit the flowsheet was not completed. Some possible solutions to this problem would be to (1) make the flowsheet an integral part of the patient's chart, (2) have all parts of the chart immediately available, or (3) use computerized medical records. PGs have great theoretic advantages for use in outcome research because they are a way to link process and outcome. However, we found that there must be a mechanism to have the PG flowsheet or data sheet available at the time outcome data are collected, which may be immediately after treatment or sew~ral months later. Further, the development of a PG must be a dynamic process. Inevitably, problems (e.g., exclusions and inefficiencies) will be encountered, and it is important that the PG be flexible to revision, if necessary. Use of the PG may seem like an "extra" step to some providers, and there will definitely be variation among providers in the level of enthusiasm and participation. However, if the PG was developed with the input and cooperation of all providers, and is perceived as a method to enhance patient care, ultimately the PG will be viewed favorably. This study has some limitations. The number of patients studied was small, and the study therefore lacks statistical power to detect small changes in process variables. Further, the location of the study at a general teaching hospital may limit its applicability to other health-care settings. In addition, this study falls short of an exhaustive attempt to identify and modify all sources of potential variation in the health-care process. However, as a pilot study it succeeds at demonstrating some positive effects of the application of a PG to an outpatient process of care.

STEWART et al. 393

Despite these potential problems, we plata to continue to use PGs as a method to both assess and standardize the process of health-care delivery, as well as provide a link to the measurement of clinical outcomes.

REFERENCES 1. DonabedianA. Explorationsin qualityassessmentand monitoring, vol 1. In: The definition of quality and approaches to its assessment. Ann Arbor, Michigan:HealthAdministrationPress; 1980. 2. GrimshawJM, Russell IT. Effect of clinical guidelineson medical practice: a systematicreview of rigorous evaluations.Lancet 1993;342:13i7-22. 3. AmericanAcademyof Otolaryngology-Headand Neck Surgery, Inc. Clinical indicators compendium. Alexandria, Virginia: AAO-HNS; 1995. 4. ManningSC, Beste C, McBride T, GoldbergA. An assessment of preoperativecoagulationscreeningfor tonsillectomyand adenoidectomy. Int J Pediatr Otorhinolaryngol1987;13:237-44. 5. SuchmanAI, MushlinAI. How well does the activated partial thromboplastin time predict postoperative hemorrhage? JAMA 1986;256:750-3. 6. Gabalski EC, Mattucci KF, Setzen M, Moleski R Ambulatory tonsillectomyand adenoidectomy.Laryngoscope 1996;106:7780. 7. ManigliaAJ, Knshner H, Cozzi L. Adenotonsillectomy:a safe outpatient procedure. Arch Otolaryngol Head Neck Surg 1989;115:92-4. 8. Reiner SA, SawyerWR Clark KF, Wood MW. Safety of outpatient tonsillectomyand adenoidectomy.OtolaryngolHead Neck Surg 1990;102:161-8. 9. Guida RA, Mattucci KF. Tonsillectomyand adenoidectomy: an outpatient or outpatient procedure? Laryngoscope 1990;100:491-3. 10. NicklansPJ, Herzon FS, SteinleEW. Short-stay outpatienttonsillectomy.Arch OtolaryngolHead Neck Surg 1995;121:521-4. 11. TelianSA, HandlerSD, FleisherGR, BaranakCC, WetmoreRF, Potsic WP. The effect of antibiotictherapy on recovery after tonsillectomy in children: a controlled study. Arch Otolaryngol Head Neck Surg 1986;112:610-5. 12. Woolf SH. Practice guidelines: a new reality in medicine, III: impact on patient care. Arch Intern Med 1993;153:2646-55. 13. Luttman RJ. The critical path method alone does nothing to improve performance.Qual Rev Bull 1993;19:142-3. t4. Lomas J, Anderson GM, Domnick-PierreK, Vayda E, Enkin MW, Hannah WJ. Do practice guidelinesguide practice? The effect of a consensusstatementon the practice of physicians.N Engl J Med 1989;321:1306-11. 15. Orldn FK. Practice standards:the Midas touch or the emperor's new clothes?Anesthesiology1989;70:567-71. 16. GoldmanL. Changingphysicians'behavior:the pot and the kettle. N Engl J Med 1990;322:1524-5. 17. Bluestone CD, Klein JO. Clinical practice guideline on otitis media with effusionin young children:strengths and weaknesses. OtolaryngolHead Neck Surg 1995;i12:507-11. 18. Eddy DM. Designing a practice policy: standards, guidelines, and options. JAMA 1990;263:3077-84. 19. Eddy DM. Practice policies: guidelines for methods. JAMA I990;263:1839-41. 20. Woolf SH. Practice guidelines: a new reality in medicine, II: methods of developing guidelines. Arch Intern Med 1992;152:946-52, 21. Eagle KA, Mulley AG, Skates SJ, Reder VA, Nicholson BW. Length of stay in the intensive care unit: effects of practice guidelinesand feedback. JAMA 1990;264:992-7.