- Email: [email protected]
Hawthorne Effect: Implications for Prehospital Research
EMS/ORIGINAL CONTRIBUTION
HawthorneEffect: Implications for Prehospital Research From the School of Medicine, Truman Medical Center*, and the School of Pharmacy, University of Missouri-Kansas City; and the Department of Health, Division of Emergency Medical Services, Kansas
City, Missouri. ~ Receivedfor publication January 27, 1995. Revision received April 28, 1995. Acceptedfor publication May 11, 1995. Presented at the Societyfor Academic Emergency Medicine Annual Meeting, Washington DC, May 1994. Funded in part by grantsfrom the Emergency Physicians Foundation of Greater Kansas City and the Missouri chapter of the American College of Emergency Physicians. Copyright © by the American College of Emergency Physicians.
Jack P Campbell, MD, FACEP*e Vada A Maxey, MD* William A Watson, PharmD**
Study objective: The phenomenon of altered behavior or performance resulting from awareness of being a part of an experimental study has been termed the "Hawthorne effect." Prehospital studies generally involve paramedics or are designed to use data collected by paramedics. Our objective was to determine whether paramedic performance, as measured by frequency of documentation, can be modified by (1) written notification of the importance of documentation, (2) written notification of a research project involving paramedic documentation, or (3) written notification of a quality-improvement audit of paramedic documentation.
Design: Prospective, sequential intervention study with five study phases.
Setting: Urban, all-advanced life support public utility model emergency medical services system with 55,000 emergency calls per year. Participants: One hundred forty-five paramedics who com-
pleted all ambulance run reports from August 1992 to May 1993.
Results: A total of 30,828 run reports was entered into the study. Baseline undocumented parameters ranged from 3.7% to 6.5%. Compared with baseline, a memo to heighten awareness (phase 2) did not alter documentation (P>.08). A medication study memo (phase 3)improved medication documentation (P=.0005) and allergies documentation (P=.037). A qualityimprovement audit memo (phase 4)improved documentation of all parameters (P<.001). Conclusion: The Hawthorne effect occurs in prehospital research. It does not require direct observation, nor does it require direct feedback. However, it may require a perceived demand for performance. The Hawthorne effect must be considered in the design of prehospital studies and interpretation of data collected by paramedics.
59O
ANNALS OF EMERGENCY MEDICINE
2 6 : 5 NOVEMBER
1995
HAWTHORNE EFFECT CamFbell, Maxey & Watson
[Campbell JP, Maxey VA, Watson WA: Hawthorne effect: Implications for prehospital research. Ann EmergMed November 1995;26:590-594.] INTRODUCTION Awareness of being observed can alter the way in which a person acts or performs. It has been suggested that a person's awareness ,of being in an experiment, through intentional or unintentional cues, causes this effect. 1,2 However, awareness alone may not always induce the effect. A demand for change in performance must also be felt by the person, or a performance feedback mechanism must be in place. 3 The phenomenon of altering behavior or performance because one is aware of being a part of an experimental study has been termed the "Hawthorne effect." The Hawthorne effect derives its name from experimental studies performed from 1927 through 1932 in the Hawthorne Works plant of the Western Electric Company in Chicago. 4 The test-room method was developed in part 1 of the study for observation of the effects of altering work conditions on employee efficiency. Despite the many work condition changes, the production of the workers continued to increase, even when they returned to the original prestudy work conditions. Usually, when the Hawthorne effect is cited this part of the entire original study is what is being referred to. The potential influence of the Hawthorne effect on emergency medical services (EMS) research and the variables that affect the Hawthorne effect have not been evaluated. Prehospital studies generally either involve observation of paramedics or are designed to use data collected by paramedics. In either situation, the paramedics are aware of the study. If awareness alters paramedic performance, the resuiting Hawthorne effect can significantly alter study results to the extent that they cannot be applied to the everyday provision of prehospital care under nonstudy conditions. Our objective was to determine whether paramedic performance, as measured on the basis of frequency of documentation, ,:an be modified by (1) written notification of the importance of documentation, (2) written notification of a research project involving paramedic documentation, or (3) written notification of a qualityimprovement audit of paramedic documentation.
MATERIALS AND METHODS We used a prospective, sequential intervention study design with a baseline (phase 1) followed by three sepa-
NOVEMBER 1995
26:5
ANNALS OF EMERGENCY MEDICINE
rate phases that introduced different study variables: a memo to heighten general documentation awareness (phase 2), a memo announcing involvement in a medication study (phase 3), and a memo announcing a qualityimprovement audit of documentation (phase 4). The memo to heighten general awareness was 11 lines long. It reminded the paramedics that the ambulance run report is a medicolegal document that affects patient care, quality-improvement activities, medicolegal issues, and system reviews. The three study parameters were not mentioned, and paramedics were encouraged to accurately complete ambulance run reports to facilitate the goal of clinical excellence. The memo announcing involvement in a medication study was seven lines long. It provided the date on which the system would begin to be involved in a study of patients' current medications and stated that the "reeds" section of the ambulance run report was to be completed for each patient. If the patient was taking no medications, the word "none" was to be entered. The memo announcing a quality-improvement audit was six lines long. It stated that the medication study had been completed and that useful information had been obtained. The paramedics were thanked for their participation. The memo then stated the date on which a quality-improvement review of ambulance run report documentation of the allergies, medications, and past medical history portions of the run reports would be started. A final g-week washout period concluded the study. During this period, no variables were introduced (phase 5). Each study phase was 7 weeks long, with 1 week separating phases. During this separation week, the variable for the next phase was introduced by means of a written memo disseminated with paychecks. The period of study was August 1992 to May 1993. The setting of the study was an urban, all-advanced life support (ALS) public utility model EMS system with 55,000 emergency calls per year and a 440-square mile service area. The participants were 145 paramedics who completed all ambulance run reports during the study period. All ambulance run reports during the study period were reviewed for completeness of documentation of three study parameters (allergies, medications, and past medical history). The parameters were a standard part of the ambulance run report. The information was manually collected and entered into a database by one person using written guidelines. The documentation of a parameter was considered complete if any worded documentation was entered by the paramedic. A yes or no response was
59 1
H A W T H O R N E EFFECT
Campbell, Maxey & Watson
recorded for each parameter completed. No feedback was given to the paramedics with regard to documentation completeness during the study We recorded the city license number of each paramedic for each run report reviewed to provide a method of assessing the effect of the study variables on individuals. We maintained paramedic confidentiality by securing the database for the paramedic city license number. No patient identifiers were entered from the run reports. All system paramedics had been involved with at least one EMS system research study or one system quality-improvement audit before participating in this study We calculated the total completion percentages for each parameter in each phase. Documentation frequencies of the three study parameters were compared with baseline values with the use of )~2 analysis to determine the effects of the variables (memos). The addition of each variable was also evaluated by means of comparison with the immediately preceding phase. We performed linear regression of weekly parameter documentation frequencies for each phase to determine the effect of time. 5 The documentation frequencies were compared for individual paramedics with at least 40 reports for each study phase. We stratified comparisons of individual paramedic performance on the basis of baseline completion frequency Natural cutoffs were determined from the baseline documentation frequencies. Documentation fieFigure 1.
Parameter completion rate for the five study phases. % ComNetedRunReports •
Allergies Medications Pastrnedical history
3 Phase
592
quencies were classified as excellent (97.7% to 100%), good (91.8% to 96.5%), and fair (75.8% to 88.0%). The study protocol was approved by the University of Missouri-Kansas City Adutt Health Sciences Institutional Review Board. RESULTS
The study included 30,828 ambulance run reports. The number of reports per week ranged from 802 to 1,073 during the 39-week study period, The results of the baseline phase demonstrated that 90% of the records were documented completely. A change from 90% to 92% (a 20% decrease in incomplete run reports) could be detected at a P value of .05, with a power of .80. Figure 1 demonstrates the completion percentages for each parameter of each phase. The memo to improve documentation awareness (phase 2) resulted in no significant change from baseline (P>__.08).The memo informing paramedics of the medication study (phase 3) improved medication (P=.0005) and allergy (P=.037) documentation. The notification of paramedics of the quality-improvement audit (phase 4) improved the documentation of all parameters (P<.001) compared with baseline (phase 1) and lasted 13 weeks. Comparison with values for the preceding phase instead of the study baseline demonstrated similar statistical changes. Phase 3 compared with phase 2 showed improvement in medication (P=.0005), allergy (P=.002), and past medical history (P=.011) documentation. Comparison of phase 4 and phase 3 also revealed improvement in the documentation of all parameters (medications, P=.0005; allergy, P.0005; past medical history, P=.017). Linear regression demonstrated no significant change by week during phases 1, 2, 4, and 5. The baseline (phase 1) was constant per week throughout the phase. The medication study memo (phase 3) showed a significant change for medication (P=.008, slope=.48) and allergy (P=.008, slope=.54), but not for past medical history (P=. 14, slope=.24). The quality-improvement audit memo (phase 4) had the maximum effect early and remained constant each week throughout the phase. Using the criteria of a minimum of 40 reports per medic per phase, 23.4% of the paramedics (34 of 145) were selected. These 34 paramedics completed more than 17,000 of the total run reports. Individual paramedics had from 40 to 179 reports per study phase. Twenty paramedics were grouped in the excellent category, eight paramedics were grouped in the good category, and six paramedics were in the fair catego W. Figure 2 demon-
ANNALS OF EMERGENCY MEDICINE 26:5 NOVEMBER 1995
HAWTHORNE EFFECT
Campbdl, Maxey & Watson
strates the completion rates of the three groups as they progressed through the five study phases. With the distribution of the awamness memo (phase 2), there was a decline in the completion rates by paramedic groups 2 and 3. The medication study memo (phase 3) caused improvement in all three groups, with the most prominent changes in groups 2 and 3. This is again demonstrated with the quality-improvement audit memo (phase 4). DISCUSSION
Data collection in EMS research is limited by many practical issues. Spaite et al6 suggested that major improvement in the understanding of EMS systems would require specific field observation. A difficulty with observation study methods is the potential impact of the Hawthorne effect. Compared with the results of retrospective studies 7,s, direct observation %~° resulted in significantly shorter time for IV-line placement. It must be considered that the participants in the prospective direct observational studies became aware of the study objective and performed better (eg, faster). To further complicate EMS study method issues, our study appears to have demonstrated the Hawthorne effect in the absence of direct, in-field paramedic observation. The addition of sequential variables resulted in continued improvement in paramedic documentation, a result similar to the original Hawthorne study's test-room results. Notification of a quality-improvement audit appears to induce the effect more than does notification of involvement in a prehospital study. Both variables imply a demand for improved performance, but the qualityimprovement audit may imply punishment if the demand is not met. This may be a result of paramedics linking system quality-improvement audits to individual quality-assessment audits. The latter can have negative implications. Until the two can be clearly differentiated within a system, these results suggest that researchers should be wary of linking studies to quality-improvment audits or conducting research during an audit phase. This may produce results not applicable to everyday events. Although the effect was demonstrated without direct observation, the duration of the effect remains unanswered. No intervention was introduced into phase 5, and completion rates remained similar to those of phase 4. Whether the completion rate would have decreased toward baseline had the quality-improvment audit phase been formally ended remains unknown. Furthermore, it may be speculated that the effect during
NOVEMBER 1995 26:5 ANNALS OF EMERGENCY MEDICrNE
the quality-improvement phase was not the result of one memo but of the stepwise addition of all variables. The medication study (phase 3) was announced as completed 'before the quality-improvement audit (phase 4) was begun. It is unknown whether announcing the end of phase 3 eliminated this stepwise addition of variables. Although no direct feedback was given the paramedics about the study, the paramedics may have perceived over time that they were under observation. However, comparisons with the preceding phase instead of the baseline did not significantly change the study findings. In addition, linear regression demonstrated no significant change by week in the phases, except phase 3 (medication study memo). The only variables that improved per week during phase 3 were those directly related to the data perceived necessary for the medication study (medication, allergy). In reviewing the documentation rates of the stratified paramedics, it must be emphasized that the baseline documentation frequency was high for groups 1 and 2. Minimal change was seen in group 1, probably because of the already outstanding documentation rates and little room for improvement. A constant improvement across phases 3 and 4 was seen for both paramedic groups 2 and 3, with the largest percentage of change occurring in group 3. Why the documentation rates declined for groups 2 and 3 during phase 2 is unknown. Limitations of the study include a selection bias for those paramedics with 40 or more run reports per phase. Figure 2.
Median percent completenessfor each of the three groups of paramedics. % Completed Run Reports 100,3, x
Excellent v Good • Fair
95
ii 80 2
3
4
5
Phase
593
HAWTHORNE
EFFECT
Campbell, Maxey & Watson
However, because these paramedics were responsible for more than half of the reports completed, the bias may be minimal. The paramedics with fewer than 40 run reports were often part-time equivalent personnel or minimal shift full-time equivalent personnel. Paramedics who took vacation or other leave during the study period did not generate the required 40 run reports for each phase and were eliminated from this part of the study. We believe this criterion was necessary for equal comparison of the paramedics across all phases. Another limitation may have arisen from the fact that only the frequency of documentation, not the accuracy of information, was studied. This was a study of performance based on documentation, and how the Hawthorne effect may influence this performance. Whether this improved documentation performance equates with improved patient care or patient outcomes remains to be studied. Finally, a sequential intervention design was used in this study. It is unclear whether the same results would have been obtained had the order been changed. To eliminate the potential stepwise addition of variables, future studies should formally begin and end each phase with a return to baseline before the next variable is introduced. The order of variable introduction could be randomly assigned. Future studies may attempt to modify or reduce the Hawthorne effect by incorporating one of the following: (1) studying each variable individually and waiting for the return to baseline before a new variable is introduced; (2) telling the paramedics that they are involved in a study, but specifically request no change in their performance; (3) randomizing the paramedics as to who will participate in the study and who will be the controls; and (4) considering using patient outcome parameters instead of documentation parameters.
REFERENCES 1. Reiss F: The Hawthorne Effect in a Pilot Program(thesis]. Glassboro, New Jersey, Glassboro State College, 1979. 2. Carlopio J: The Hawthorne Effect: Awareness of Experimental Participation or Expen'rnental Demands?(thesis). Winnipeg, Manitoba, Canada,University of Manitoba, 1982. 3. ParsonsHM: Hawthorne Effect" A More Adequate Explanation (cassette). Los Angeles, Development Digest (published by CREDRCorporation),1974. 4. Roethlisberger FJ, DicksonW J: Management and the Worker; An Account of a Research Program Conductedby Western Electric Company,Hawthorne Works, Chicago.Cambridge, Massachusetts: Harvard University Press. 1939. 5. Glantz SA: PrirnerofBiostatistics, ed 3. New York, McGraw-Hill, 1992, pp 218-277. 6. Spaite DW, ValenzuelaTD, Meislin HW: Barriers to EMS system evaluation, problems associated with field data collection. PrehospDisaster Med1993;8:S35-S40. 7. McSwain GR, GarrisonWB, Artz CP: Evaluation of resuscitation from cardiepulmonary arrest by paramedics.Ann ErnergMed 1980;9:341-345. 8. Smith JP, Bodai BI, Hill AS, et al: Prehospital stabilization of critically injured patients: A failed concept. J Trauma1985;25:65-70. 9. Jones SE, Nesper TP, Alcouloumre E: Prehospital intravenous line placement: A prospective study. Ann ErnergMed 1989;18:244-246. 10. Pons PT, Moore EE, CusickJM, et el: Prehospitalvenous access in an urban paramedic system: A prospective an-scene analysis. J Trauma1988;28:1480-146&
Reprint no. 47/1/67913 Address for reprints: JackPCampbell,MD, FACEP Departmentof EmergencyMedicine TrumanMedicalCenter 2301 Holmes KansasCity, Missouri64108 816-556-3250 Fax816-881-6282
CONCLUSION
The Hawthorne effect occurs in prehospital research. It does not require direct observation, nor does it require direct feedback. However, it may require a perceived demand for performance. The addition of sequential variables produced an improvement in performance similar to that of the original Hawthorne study's test-room method. Paramedic awareness of research requiring performance or concurrent quality-improvement audits may produce data that are not representative of nonstudy conditions. The Hawthorne effect must be considered in the design of prohospital studies and the interpretation of data collected by paramedics.
59 4
ANNALS OF EMERGENCY MEDICINE
26:5
NOVEMBER 1995
HawthorneEffect: Implications for Prehospital Research From the School of Medicine, Truman Medical Center*, and the School of Pharmacy, University of Missouri-Kansas City; and the Department of Health, Division of Emergency Medical Services, Kansas
City, Missouri. ~ Receivedfor publication January 27, 1995. Revision received April 28, 1995. Acceptedfor publication May 11, 1995. Presented at the Societyfor Academic Emergency Medicine Annual Meeting, Washington DC, May 1994. Funded in part by grantsfrom the Emergency Physicians Foundation of Greater Kansas City and the Missouri chapter of the American College of Emergency Physicians. Copyright © by the American College of Emergency Physicians.
Jack P Campbell, MD, FACEP*e Vada A Maxey, MD* William A Watson, PharmD**
Study objective: The phenomenon of altered behavior or performance resulting from awareness of being a part of an experimental study has been termed the "Hawthorne effect." Prehospital studies generally involve paramedics or are designed to use data collected by paramedics. Our objective was to determine whether paramedic performance, as measured by frequency of documentation, can be modified by (1) written notification of the importance of documentation, (2) written notification of a research project involving paramedic documentation, or (3) written notification of a quality-improvement audit of paramedic documentation.
Design: Prospective, sequential intervention study with five study phases.
Setting: Urban, all-advanced life support public utility model emergency medical services system with 55,000 emergency calls per year. Participants: One hundred forty-five paramedics who com-
pleted all ambulance run reports from August 1992 to May 1993.
Results: A total of 30,828 run reports was entered into the study. Baseline undocumented parameters ranged from 3.7% to 6.5%. Compared with baseline, a memo to heighten awareness (phase 2) did not alter documentation (P>.08). A medication study memo (phase 3)improved medication documentation (P=.0005) and allergies documentation (P=.037). A qualityimprovement audit memo (phase 4)improved documentation of all parameters (P<.001). Conclusion: The Hawthorne effect occurs in prehospital research. It does not require direct observation, nor does it require direct feedback. However, it may require a perceived demand for performance. The Hawthorne effect must be considered in the design of prehospital studies and interpretation of data collected by paramedics.
59O
ANNALS OF EMERGENCY MEDICINE
2 6 : 5 NOVEMBER
1995
HAWTHORNE EFFECT CamFbell, Maxey & Watson
[Campbell JP, Maxey VA, Watson WA: Hawthorne effect: Implications for prehospital research. Ann EmergMed November 1995;26:590-594.] INTRODUCTION Awareness of being observed can alter the way in which a person acts or performs. It has been suggested that a person's awareness ,of being in an experiment, through intentional or unintentional cues, causes this effect. 1,2 However, awareness alone may not always induce the effect. A demand for change in performance must also be felt by the person, or a performance feedback mechanism must be in place. 3 The phenomenon of altering behavior or performance because one is aware of being a part of an experimental study has been termed the "Hawthorne effect." The Hawthorne effect derives its name from experimental studies performed from 1927 through 1932 in the Hawthorne Works plant of the Western Electric Company in Chicago. 4 The test-room method was developed in part 1 of the study for observation of the effects of altering work conditions on employee efficiency. Despite the many work condition changes, the production of the workers continued to increase, even when they returned to the original prestudy work conditions. Usually, when the Hawthorne effect is cited this part of the entire original study is what is being referred to. The potential influence of the Hawthorne effect on emergency medical services (EMS) research and the variables that affect the Hawthorne effect have not been evaluated. Prehospital studies generally either involve observation of paramedics or are designed to use data collected by paramedics. In either situation, the paramedics are aware of the study. If awareness alters paramedic performance, the resuiting Hawthorne effect can significantly alter study results to the extent that they cannot be applied to the everyday provision of prehospital care under nonstudy conditions. Our objective was to determine whether paramedic performance, as measured on the basis of frequency of documentation, ,:an be modified by (1) written notification of the importance of documentation, (2) written notification of a research project involving paramedic documentation, or (3) written notification of a qualityimprovement audit of paramedic documentation.
MATERIALS AND METHODS We used a prospective, sequential intervention study design with a baseline (phase 1) followed by three sepa-
NOVEMBER 1995
26:5
ANNALS OF EMERGENCY MEDICINE
rate phases that introduced different study variables: a memo to heighten general documentation awareness (phase 2), a memo announcing involvement in a medication study (phase 3), and a memo announcing a qualityimprovement audit of documentation (phase 4). The memo to heighten general awareness was 11 lines long. It reminded the paramedics that the ambulance run report is a medicolegal document that affects patient care, quality-improvement activities, medicolegal issues, and system reviews. The three study parameters were not mentioned, and paramedics were encouraged to accurately complete ambulance run reports to facilitate the goal of clinical excellence. The memo announcing involvement in a medication study was seven lines long. It provided the date on which the system would begin to be involved in a study of patients' current medications and stated that the "reeds" section of the ambulance run report was to be completed for each patient. If the patient was taking no medications, the word "none" was to be entered. The memo announcing a quality-improvement audit was six lines long. It stated that the medication study had been completed and that useful information had been obtained. The paramedics were thanked for their participation. The memo then stated the date on which a quality-improvement review of ambulance run report documentation of the allergies, medications, and past medical history portions of the run reports would be started. A final g-week washout period concluded the study. During this period, no variables were introduced (phase 5). Each study phase was 7 weeks long, with 1 week separating phases. During this separation week, the variable for the next phase was introduced by means of a written memo disseminated with paychecks. The period of study was August 1992 to May 1993. The setting of the study was an urban, all-advanced life support (ALS) public utility model EMS system with 55,000 emergency calls per year and a 440-square mile service area. The participants were 145 paramedics who completed all ambulance run reports during the study period. All ambulance run reports during the study period were reviewed for completeness of documentation of three study parameters (allergies, medications, and past medical history). The parameters were a standard part of the ambulance run report. The information was manually collected and entered into a database by one person using written guidelines. The documentation of a parameter was considered complete if any worded documentation was entered by the paramedic. A yes or no response was
59 1
H A W T H O R N E EFFECT
Campbell, Maxey & Watson
recorded for each parameter completed. No feedback was given to the paramedics with regard to documentation completeness during the study We recorded the city license number of each paramedic for each run report reviewed to provide a method of assessing the effect of the study variables on individuals. We maintained paramedic confidentiality by securing the database for the paramedic city license number. No patient identifiers were entered from the run reports. All system paramedics had been involved with at least one EMS system research study or one system quality-improvement audit before participating in this study We calculated the total completion percentages for each parameter in each phase. Documentation frequencies of the three study parameters were compared with baseline values with the use of )~2 analysis to determine the effects of the variables (memos). The addition of each variable was also evaluated by means of comparison with the immediately preceding phase. We performed linear regression of weekly parameter documentation frequencies for each phase to determine the effect of time. 5 The documentation frequencies were compared for individual paramedics with at least 40 reports for each study phase. We stratified comparisons of individual paramedic performance on the basis of baseline completion frequency Natural cutoffs were determined from the baseline documentation frequencies. Documentation fieFigure 1.
Parameter completion rate for the five study phases. % ComNetedRunReports •
Allergies Medications Pastrnedical history
3 Phase
592
quencies were classified as excellent (97.7% to 100%), good (91.8% to 96.5%), and fair (75.8% to 88.0%). The study protocol was approved by the University of Missouri-Kansas City Adutt Health Sciences Institutional Review Board. RESULTS
The study included 30,828 ambulance run reports. The number of reports per week ranged from 802 to 1,073 during the 39-week study period, The results of the baseline phase demonstrated that 90% of the records were documented completely. A change from 90% to 92% (a 20% decrease in incomplete run reports) could be detected at a P value of .05, with a power of .80. Figure 1 demonstrates the completion percentages for each parameter of each phase. The memo to improve documentation awareness (phase 2) resulted in no significant change from baseline (P>__.08).The memo informing paramedics of the medication study (phase 3) improved medication (P=.0005) and allergy (P=.037) documentation. The notification of paramedics of the quality-improvement audit (phase 4) improved the documentation of all parameters (P<.001) compared with baseline (phase 1) and lasted 13 weeks. Comparison with values for the preceding phase instead of the study baseline demonstrated similar statistical changes. Phase 3 compared with phase 2 showed improvement in medication (P=.0005), allergy (P=.002), and past medical history (P=.011) documentation. Comparison of phase 4 and phase 3 also revealed improvement in the documentation of all parameters (medications, P=.0005; allergy, P.0005; past medical history, P=.017). Linear regression demonstrated no significant change by week during phases 1, 2, 4, and 5. The baseline (phase 1) was constant per week throughout the phase. The medication study memo (phase 3) showed a significant change for medication (P=.008, slope=.48) and allergy (P=.008, slope=.54), but not for past medical history (P=. 14, slope=.24). The quality-improvement audit memo (phase 4) had the maximum effect early and remained constant each week throughout the phase. Using the criteria of a minimum of 40 reports per medic per phase, 23.4% of the paramedics (34 of 145) were selected. These 34 paramedics completed more than 17,000 of the total run reports. Individual paramedics had from 40 to 179 reports per study phase. Twenty paramedics were grouped in the excellent category, eight paramedics were grouped in the good category, and six paramedics were in the fair catego W. Figure 2 demon-
ANNALS OF EMERGENCY MEDICINE 26:5 NOVEMBER 1995
HAWTHORNE EFFECT
Campbdl, Maxey & Watson
strates the completion rates of the three groups as they progressed through the five study phases. With the distribution of the awamness memo (phase 2), there was a decline in the completion rates by paramedic groups 2 and 3. The medication study memo (phase 3) caused improvement in all three groups, with the most prominent changes in groups 2 and 3. This is again demonstrated with the quality-improvement audit memo (phase 4). DISCUSSION
Data collection in EMS research is limited by many practical issues. Spaite et al6 suggested that major improvement in the understanding of EMS systems would require specific field observation. A difficulty with observation study methods is the potential impact of the Hawthorne effect. Compared with the results of retrospective studies 7,s, direct observation %~° resulted in significantly shorter time for IV-line placement. It must be considered that the participants in the prospective direct observational studies became aware of the study objective and performed better (eg, faster). To further complicate EMS study method issues, our study appears to have demonstrated the Hawthorne effect in the absence of direct, in-field paramedic observation. The addition of sequential variables resulted in continued improvement in paramedic documentation, a result similar to the original Hawthorne study's test-room results. Notification of a quality-improvement audit appears to induce the effect more than does notification of involvement in a prehospital study. Both variables imply a demand for improved performance, but the qualityimprovement audit may imply punishment if the demand is not met. This may be a result of paramedics linking system quality-improvement audits to individual quality-assessment audits. The latter can have negative implications. Until the two can be clearly differentiated within a system, these results suggest that researchers should be wary of linking studies to quality-improvment audits or conducting research during an audit phase. This may produce results not applicable to everyday events. Although the effect was demonstrated without direct observation, the duration of the effect remains unanswered. No intervention was introduced into phase 5, and completion rates remained similar to those of phase 4. Whether the completion rate would have decreased toward baseline had the quality-improvment audit phase been formally ended remains unknown. Furthermore, it may be speculated that the effect during
NOVEMBER 1995 26:5 ANNALS OF EMERGENCY MEDICrNE
the quality-improvement phase was not the result of one memo but of the stepwise addition of all variables. The medication study (phase 3) was announced as completed 'before the quality-improvement audit (phase 4) was begun. It is unknown whether announcing the end of phase 3 eliminated this stepwise addition of variables. Although no direct feedback was given the paramedics about the study, the paramedics may have perceived over time that they were under observation. However, comparisons with the preceding phase instead of the baseline did not significantly change the study findings. In addition, linear regression demonstrated no significant change by week in the phases, except phase 3 (medication study memo). The only variables that improved per week during phase 3 were those directly related to the data perceived necessary for the medication study (medication, allergy). In reviewing the documentation rates of the stratified paramedics, it must be emphasized that the baseline documentation frequency was high for groups 1 and 2. Minimal change was seen in group 1, probably because of the already outstanding documentation rates and little room for improvement. A constant improvement across phases 3 and 4 was seen for both paramedic groups 2 and 3, with the largest percentage of change occurring in group 3. Why the documentation rates declined for groups 2 and 3 during phase 2 is unknown. Limitations of the study include a selection bias for those paramedics with 40 or more run reports per phase. Figure 2.
Median percent completenessfor each of the three groups of paramedics. % Completed Run Reports 100,3, x
Excellent v Good • Fair
95
ii 80 2
3
4
5
Phase
593
HAWTHORNE
EFFECT
Campbell, Maxey & Watson
However, because these paramedics were responsible for more than half of the reports completed, the bias may be minimal. The paramedics with fewer than 40 run reports were often part-time equivalent personnel or minimal shift full-time equivalent personnel. Paramedics who took vacation or other leave during the study period did not generate the required 40 run reports for each phase and were eliminated from this part of the study. We believe this criterion was necessary for equal comparison of the paramedics across all phases. Another limitation may have arisen from the fact that only the frequency of documentation, not the accuracy of information, was studied. This was a study of performance based on documentation, and how the Hawthorne effect may influence this performance. Whether this improved documentation performance equates with improved patient care or patient outcomes remains to be studied. Finally, a sequential intervention design was used in this study. It is unclear whether the same results would have been obtained had the order been changed. To eliminate the potential stepwise addition of variables, future studies should formally begin and end each phase with a return to baseline before the next variable is introduced. The order of variable introduction could be randomly assigned. Future studies may attempt to modify or reduce the Hawthorne effect by incorporating one of the following: (1) studying each variable individually and waiting for the return to baseline before a new variable is introduced; (2) telling the paramedics that they are involved in a study, but specifically request no change in their performance; (3) randomizing the paramedics as to who will participate in the study and who will be the controls; and (4) considering using patient outcome parameters instead of documentation parameters.
REFERENCES 1. Reiss F: The Hawthorne Effect in a Pilot Program(thesis]. Glassboro, New Jersey, Glassboro State College, 1979. 2. Carlopio J: The Hawthorne Effect: Awareness of Experimental Participation or Expen'rnental Demands?(thesis). Winnipeg, Manitoba, Canada,University of Manitoba, 1982. 3. ParsonsHM: Hawthorne Effect" A More Adequate Explanation (cassette). Los Angeles, Development Digest (published by CREDRCorporation),1974. 4. Roethlisberger FJ, DicksonW J: Management and the Worker; An Account of a Research Program Conductedby Western Electric Company,Hawthorne Works, Chicago.Cambridge, Massachusetts: Harvard University Press. 1939. 5. Glantz SA: PrirnerofBiostatistics, ed 3. New York, McGraw-Hill, 1992, pp 218-277. 6. Spaite DW, ValenzuelaTD, Meislin HW: Barriers to EMS system evaluation, problems associated with field data collection. PrehospDisaster Med1993;8:S35-S40. 7. McSwain GR, GarrisonWB, Artz CP: Evaluation of resuscitation from cardiepulmonary arrest by paramedics.Ann ErnergMed 1980;9:341-345. 8. Smith JP, Bodai BI, Hill AS, et al: Prehospital stabilization of critically injured patients: A failed concept. J Trauma1985;25:65-70. 9. Jones SE, Nesper TP, Alcouloumre E: Prehospital intravenous line placement: A prospective study. Ann ErnergMed 1989;18:244-246. 10. Pons PT, Moore EE, CusickJM, et el: Prehospitalvenous access in an urban paramedic system: A prospective an-scene analysis. J Trauma1988;28:1480-146&
Reprint no. 47/1/67913 Address for reprints: JackPCampbell,MD, FACEP Departmentof EmergencyMedicine TrumanMedicalCenter 2301 Holmes KansasCity, Missouri64108 816-556-3250 Fax816-881-6282
CONCLUSION
The Hawthorne effect occurs in prehospital research. It does not require direct observation, nor does it require direct feedback. However, it may require a perceived demand for performance. The addition of sequential variables produced an improvement in performance similar to that of the original Hawthorne study's test-room method. Paramedic awareness of research requiring performance or concurrent quality-improvement audits may produce data that are not representative of nonstudy conditions. The Hawthorne effect must be considered in the design of prohospital studies and the interpretation of data collected by paramedics.
59 4
ANNALS OF EMERGENCY MEDICINE
26:5
NOVEMBER 1995