- Email: [email protected]
Out-of-Hospital Violence Injury Surveillance: Quality of Data Collection
EMS/BRIEF REPORT
Out-of-Hospital Violence Injury Surveillance: Quality of Data Collection
From HealthSpan Transportation Services,* Regional Injury Prevention Research Center and Center for Violence Prevention and Control, Division of Environmental and Occupational Health,‡ Division of Biostatistics,§ School of Public Health, University of Minnesota, Minneapolis, MN.
Lynn A Boergerhoff, MPH, EMT-P* Susan Goodwin Gerberich, PhD‡ Aparna Anderson, PhD§ Laura Kochevar, PhD‡ Lance Waller, PhD§
Received for publication November 20, 1998. Revision received August 5, 1999. Accepted for publication September 13, 1999. Supported, in part, by the Allina Foundation; Allina Hospitals & Clinics Medical Transportation, Allina Health System; the Center for Violence Prevention and Control, Division of Environmental and Occupational Health, School of Public Health, University of Minnesota; and the Minnesota Department of Health. Address for reprints: Anita Berg, Director of Research and Quality, Allina Hospitals & Clinics Medical Transportation, 167 Grand Avenue, St. Paul, MN 55102; 612-228-8407, fax 612-228-8484; E-mail [email protected]. Copyright © 1999 by the American College of Emergency Physicians. 0196-0644/99/$8.00 + 0 47/1/102681
Study objective: The recognized need to improve data collection for violence prevention may be met, in part, by using outof-hospital data for injury surveillance. The purpose of the Prehospital Violence Injury Surveillance project was to examine the extent to which paramedics can adequately collect information about injuries, particularly intentional injuries, at emergency scenes. Methods: Paramedics in a large Midwestern metropolitan area were trained to assess violence-related events and collect relevant data using a modified ambulance run report form. Data collected from 8 violence-related training scenarios and from 13 ride-along observations were analyzed to estimate paramedic interrater reliability using the κ statistic. Data from 7,363 run report forms, filed during a 3-month study period, were abstracted and analyzed for completeness and quality. Results: Paramedics demonstrated fair to good, and sometimes excellent, interrater agreement when documenting the training scenarios. Paramedics revealed barriers to collecting violence-related out-of-hospital data. The paramedics and the observer disagreed in documenting 77% of the ride-along observations. Overall, 73% of abstracted run report forms showed documentation errors, with more than 99% of these reports containing errors of omission and 29% showing internal documentation inconsistencies. Despite the emphasis on violence-related data, documentation of domestic abuse screening was missing from more than 99% of run reports from female patients. Conclusion: Significant barriers to quality out-of-hospital data collection were identified during study implementation and in abstracted run reports. These barriers included the following: lack of organizational support; characteristics of the violence-related data elements; design of the ambulance run report form; and paramedic knowledge, attitudes, and behaviors regarding data collection.
DECEMBER 1999
34:6
ANNALS OF EMERGENCY MEDICINE
7 4 5
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
[Boergerhoff LA, Gerberich SG, Anderson A, Kochevar L, Waller L: Out-of-hospital violence injury surveillance: Quality of data collection. Ann Emerg Med December 1999;34:745-750.] INTRODUCTION
The need for improved data collection in injury prevention research has been recognized in national initiatives1,2 and by emergency medical services (EMS) leaders.3,4 Because paramedics and other emergency medical technicians (EMTs) respond to violence in their communities, they offer a unique opportunity to help identify the characteristics of the violent events and the persons affected. This is particularly important when victims of violence refuse ambulance transport and avoid subsequent medical care in the hospital emergency department. Out-of-hospital data collection has been problematic for reasons including the following: uncertainty about which data variables to collect; unavailability of desired data variables in the field; difficulties implementing and maintaining out-of-hospital data collection systems; the subjective nature of some out-of-hospital assessments; inadequate data collection instruments; and human factors such as disinterest, poor understanding, and lack of compliance with regard to data collection.5 Public health surveillance is the systematic and ongoing assessment of community health status through data collection, analysis, and interpretation. The purpose of the Prehospital Violence Injury Surveillance project was to examine the extent to which paramedics can adequately collect information about injuries, particularly intentional injuries, at emergency scenes. We present an assessment of paramedic interrater reliability and of the completeness and internal consistency of out-of-hospital data collection during a 3-month study period. M AT E R I A L S A N D M E T H O D S
Approval to conduct this study, under federal guidelines for the protection of human subjects in research, was granted by the authorized institutional review board. This study was conducted in a large medical transportation company in which approximately 140 paramedics respond to more than 44,000 emergency requests per year from an estimated resident population of 675,000 persons in a major Midwestern metropolitan area. Data were collected from ambulance run report forms that were filed on patients assessed by paramedics between November 1, 1996, and January 31, 1997.
7 4 6
A team of researchers reviewed and selected data elements from among a nationally recommended data set6 already included in the existing run report form that enabled a narrative description of the patient’s problem, medical history, physical examination, and treatments; patient demographics, vital signs, Glasgow Coma Scale scores, and billing information were also included. This form was then redesigned and expanded with additional variables thought to capture important characteristics of violence. These additions included the Injury Site/Type Matrix (ISTM), in which the paramedic coded the type of injury for a body site, and the Patient Assessment Profile (PAP), used to identify violence-related characteristics of the emergency event and the persons involved (Figure). Before implementing this effort, paramedics attended mandatory 4-hour violence-related training classes approximately 6 months before the study period to acquire assessment skills for violence-related injuries and screening of female patients for history and risk of domestic violence. The purpose of the study and the out-of-hospital data collection methods and protocols were also presented in this training. Paramedics used the redesigned run report form to document 3 classroom enactments and 5 written violence-related scenarios, which were later analyzed for interrater reliability. Paramedics began using the modified ambulance run report form in October and continued its use after completion of the study. To assist paramedics with data collection, a reference book was developed and placed in each ambulance before initiation of the study. Reminders, promoting data collection methods and protocols, were placed in the company’s newsletter before and during the study. Using the PAP, paramedics were to first categorize all patients into 1 of 3 problem types: injury, possible injury, or medical. Injuries were defined as signs of acute damage to body tissues, detectable by the paramedic using common field assessment methods. Possible, or suspected, injuries were those with a mechanism of injury and symptoms (eg, pain) yet lacking signs of injury. Medical problems included all other noninjury events. If the patient had an injury or possible injury, the paramedic was to collect subsequent data variables appropriate to the situation. Specifically, female subjects were to be asked about the history or risk of domestic violence, and those between the ages of 10 and 60 with injuries or possible injuries were to be asked about possible pregnancy. The paramedic was to indicate the source of injury, weapon used, or both, and determine from the most reliable information at the emergency scene whether the injury was intentional. If so, the event was considered to be violence-
ANNALS OF EMERGENCY MEDICINE
34:6 DECEMBER 1999
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
related and was subsequently assigned International Classification of Diseases, Ninth Revision E-codes 950-958 if suicide or self-inflicted injury and 960-968 if homicide or injury purposely inflicted by another person. 7 If another person caused the patient’s injuries, the
paramedic was to document that person’s relationship to the patient. Using decision rules, the study’s principal investigator abstracted each run report form to assess completeness and internal consistency and assigned to each report form
Figure.
The Injury Site/Type Matrix and the Patient Assessment Profile.
DECEMBER 1999
34:6
ANNALS OF EMERGENCY MEDICINE
7 4 7
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
1 of 16 possible 3-digit audit scores (Table). The audit score was a composite of 3 individual numeric codes (either “1” for error or “0” for no error) representing the combined responses to each of 3 questions about the run report form’s completeness and internal validity. For example, an audit score of 110 indicated that 2 types of errors occurred on the same report form: data were missing from the narrative (the first “1”), and the narrative contained information missing from the ISTM or PAP (the second “1”), while the ISTM and PAP did not contain inconsistencies between them or with the narrative (“0” in the third digit). To protect patient confidentiality, all personal identifiers (name, home address, home phone number, and Social Security number) were removed and a unique number was used to identify each ambulance run report form. The paramedics’ interrater reliability for each of the 3 enacted and 5 written scenarios from the training sessions was assessed using the κ statistic. κ Values greater than 0.75 indicate excellent interrater agreement, those between 0.4 and 0.75 indicate fair to good agreement, and values less than 0.4 indicate poor agreement.8 Second, the study’s principal investigator, a paramedic, rode with paramedics on 13 emergency calls during the study period and filled out independent run report forms that were later compared with the paramedics’ documentation. The completeness and reliability of the paramedics’ documentation was assessed through analysis of the audit codes assigned to each abstracted report form. Statistical analysis was performed using SAS (version 6.12, 1997, SAS Institute, Inc, Cary, NC) and Splus Table.
Components of the 3-digit audit code. Audit Code Digit First
Second
Third
7 4 8
Interpretation Was information missing from the narrative and from either the ISTM or PAP? 0=no (no missing data) 1=yes (missing data) Was information present in the narrative and missing from either the ISTM or PAP? 0=no (no missing data in ISTM or PAP) 1=yes (missing data in ISTM or PAP) Was any information inconsistent between narrative and ISTM and PAP? 0=no (no inconsistency) 1=yes (inconsistency in documentation between narrative and ISTM or PAP) 2=yes (new data in ISTM or PAP not in narrative) 3=yes (inconsistency and new data in ISTM or PAP not in narrative)
software (version Splus 3.4 for Unix, 1991, Statistical Sciences, Inc, Splus, Seattle, WA). R E S U LT S
Analysis of the data from the training, involving 3 enacted and 5 written classroom scenarios, was performed by obtaining the κ values for each scenario. The κ values ranged from 0.537 (95% confidence interval [CI] 0.533 to 0.541) to 0.937 (95% CI 0.933 to 0.941). The κ values indicated paramedic interrater reliability above 0.4 in all scenarios, and above 0.75, indicating excellent agreement, in 2 scenarios. The most frequent errors were missing data and inconsistent documentation of injuries between the narrative and the ISTM. Class discussion and evaluations revealed the paramedics’ dislike of additional data collection and skepticism of out-of-hospital data used to help monitor community violence. Many paramedics saw violence as a law enforcement problem and of no interest to EMS personnel beyond emergency patient care. Paramedics identified barriers to violence-related data collection including personal discomfort with pregnancy and violence screening questions, fear of legal liability for documenting injury intent and victim-perpetrator relationships, disapproval of a violence-related patient’s personal behaviors, and distrust of certain patient self-reported data. A comparison of the documentation between the paramedics and the principal investigator during the 13 ride-along observations was conducted. Among these 13 emergency events documented, there was disagreement on 10 (77%) of the report forms. Subsequently, an analysis was conducted of the 7,363 abstracted run report forms from independent emergency events that were filed by 133 paramedics. Of these patients, 4,020 (55%) were female, 3,319 (45%) patients were male, and 24 (0.3%) had no recorded gender. The patients’ ages ranged from younger than 1 year to 103 years (mean 49 years). Medical problems occurred in 4,262 (58%) patients, whereas 1,528 (21%) patients were injured, and 1,573 (21%) patients showed possible injuries. Of the 3,101 combined injuryrelated events, 2,685 (87%) were unintentional and 416 (13%) were considered intentional and violence-related, either self-inflicted or interpersonal. Among all 7,363 run report forms abstracted, 5,353 (73%) contained at least 1 documentation error in the data elements of interest. Of run report forms with errors, 99% contained errors of omission and 29% showed inconsistencies either between the narrative and the ISTM or PAP or between the ISTM and the PAP. More than half (56%) of all run report forms contained errors of more
ANNALS OF EMERGENCY MEDICINE
34:6 DECEMBER 1999
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
than one type or in multiple locations. Notably, documentation of violence screening of the 4,020 female patients was missing from more than 99% of the corresponding run report forms. The Glasgow Coma Scale score was missing on 28% of all run report forms. Complete vital signs (systolic blood pressure, pulse, and respiratory rate) were missing on 15% of all report forms, similar to vital sign omissions of approximately 17% previously reported. 9 Of 1,568 run report forms with documentation inconsistencies, 473 (30%) contained only inconsistencies between the narrative and the ISTM or PAP, 706 (45%) of the report forms showed new data collected in the ISTM or PAP that were not documented in the narrative, and 389 (25%) contained both inconsistencies and new data in the ISTM or PAP. Thus, 70% of all injuryrelated documentation inconsistencies reflected new data collected using the fill-in bubble format that were not otherwise documented in the narrative. Overall, injury-related run report forms were more likely to contain documentation errors than were those from medical events. Of the 3,101 run report forms from injury and possible injury events, 80% omitted data elements of interest from the narrative, 99% from the ISTM or PAP, and 47% from both the narrative and the ISTM and PAP. Notably, injury intent was missing from 72% of all injury-related run report forms, and the pregnancy status of females was missing on 66% of the corresponding report forms. DISCUSSION
This study of out-of-hospital data collection for violence injury surveillance has shown that paramedics are capable of good and sometimes excellent interrater agreement when documenting enacted and written class-room scenarios, using a structured reporting format such as the ISTM and the PAP. However, this study has also revealed widespread errors of omission and inconsistency in actual out-of-hospital data collection, particularly in data elements important to violence injury surveillance. Out-of-hospital injury surveillance relies on data collection previously recognized as both problematic and promising. Significant barriers to out-of-hospital data collection for violence injury surveillance were found and revealed as limitations in 3 areas: organizational support; data elements and collection methods; and paramedic knowledge, attitudes, and behaviors.
DECEMBER 1999
34:6
ANNALS OF EMERGENCY MEDICINE
Organizational support for this project was difficult to establish and maintain. The project was not integrated into new employee orientation, continuing education, or quality assurance programs, and paramedics interpreted this data collection as optional. A previous study indicated that strong organizational support has been shown to improve out-of-hospital data collection. 10 Reliance on the narrative format, associated with the original portion of the run report form, is problematic for quality data collection because documentation practices may vary widely among paramedics. For example, in the current study, paramedics used more than 600 different abbreviations, symbols, and acronyms to describe components of the patient’s history, assessment, and treatment. The check box or fill-in bubble format, specifically designed for this effort, may improve data clarity and abstracting efficiency and may actually result in collection of data otherwise omitted from the narrative description, particularly when numerous, discrete data elements are required. Finally, paramedic knowledge, attitudes, and behaviors regarding data collection in the relatively unsupervised out-of-hospital setting may significantly limit this opportunity to collect data that can contribute to a more comprehensive understanding and prevention of community violence. The paramedics’ reluctance to collect specific violence-related data elements, particularly regarding domestic abuse screening, female pregnancy status, and injury intent, suggests that these are new and sensitive areas of questioning. Yet, they are essential data elements in a prehospital violence injury surveillance system. We thank Jon Roesler, MS, senior epidemiologist, Injury and Violence Prevention Unit, Minnesota Department of Health, for his consultation in injury surveillance design.
REFERENCES 1. US Department of Health and Human Services: Healthy People 2000: National Health Promotion and Disease Prevention Objectives: Full Report, With Commentary. Washington, DC: Government Printing Office, 1990:242-244. 2. US Department of Health and Human Services: Position Papers from The Third National Injury Control Conference. Washington, DC: US Department of Health and Human Services, 1991:159-250. 3. US National Highway Traffic Safety Administration: Emergency Medical Services Agenda for the Future [DOT HS 808 142]. Washington, DC: Government Printing Office, 1996. (TD 8.2:EM 3/18) 4. Garrison HG, Foltin GL, Becker LR, et al: The role of emergency medical services in primary injury prevention. Ann Emerg Med 1997;30:4-91. 5. Spaite DW, Valenzuela TD, Meislin HW: Barriers to clinical evaluations of EMS systems: Problems associated with field data collection, in Fitch JJ (ed): Prehospital Care Administration: Issues, Readings, Cases. Carlsbad, CA: JEMS Publishing, 1995:533-543.
7 4 9
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
6. Uniform Pre-hospital Emergency Medical Services (EMS) Conference: Dean JM (ed): Final Report. Washington, DC: US Department of Transportation, 1994. 7. International Classification of Diseases, 9th Revision— Clinical Modification, ed 4. Los Angeles, CA: Practice Management Information Corporation, 1995:522-525. 8. Fleiss JL: Statistical Methods for Rates and Proportions, ed 2. New York: John Wiley & Sons, 1981:218. 9. Moss RL: Vital signs records omissions on prehospital patient encounter forms. Prehospital Disaster Med 1993;8:21-27. 10. Spaite DW, Hanion T, Criss ED, et al: Prehospital data entry compliance by paramedics after institution of a comprehensive EMS data collection tool. Ann Emerg Med 1990;19:12701273.
7 5 0
ANNALS OF EMERGENCY MEDICINE
34:6 DECEMBER 1999
Out-of-Hospital Violence Injury Surveillance: Quality of Data Collection
From HealthSpan Transportation Services,* Regional Injury Prevention Research Center and Center for Violence Prevention and Control, Division of Environmental and Occupational Health,‡ Division of Biostatistics,§ School of Public Health, University of Minnesota, Minneapolis, MN.
Lynn A Boergerhoff, MPH, EMT-P* Susan Goodwin Gerberich, PhD‡ Aparna Anderson, PhD§ Laura Kochevar, PhD‡ Lance Waller, PhD§
Received for publication November 20, 1998. Revision received August 5, 1999. Accepted for publication September 13, 1999. Supported, in part, by the Allina Foundation; Allina Hospitals & Clinics Medical Transportation, Allina Health System; the Center for Violence Prevention and Control, Division of Environmental and Occupational Health, School of Public Health, University of Minnesota; and the Minnesota Department of Health. Address for reprints: Anita Berg, Director of Research and Quality, Allina Hospitals & Clinics Medical Transportation, 167 Grand Avenue, St. Paul, MN 55102; 612-228-8407, fax 612-228-8484; E-mail [email protected]. Copyright © 1999 by the American College of Emergency Physicians. 0196-0644/99/$8.00 + 0 47/1/102681
Study objective: The recognized need to improve data collection for violence prevention may be met, in part, by using outof-hospital data for injury surveillance. The purpose of the Prehospital Violence Injury Surveillance project was to examine the extent to which paramedics can adequately collect information about injuries, particularly intentional injuries, at emergency scenes. Methods: Paramedics in a large Midwestern metropolitan area were trained to assess violence-related events and collect relevant data using a modified ambulance run report form. Data collected from 8 violence-related training scenarios and from 13 ride-along observations were analyzed to estimate paramedic interrater reliability using the κ statistic. Data from 7,363 run report forms, filed during a 3-month study period, were abstracted and analyzed for completeness and quality. Results: Paramedics demonstrated fair to good, and sometimes excellent, interrater agreement when documenting the training scenarios. Paramedics revealed barriers to collecting violence-related out-of-hospital data. The paramedics and the observer disagreed in documenting 77% of the ride-along observations. Overall, 73% of abstracted run report forms showed documentation errors, with more than 99% of these reports containing errors of omission and 29% showing internal documentation inconsistencies. Despite the emphasis on violence-related data, documentation of domestic abuse screening was missing from more than 99% of run reports from female patients. Conclusion: Significant barriers to quality out-of-hospital data collection were identified during study implementation and in abstracted run reports. These barriers included the following: lack of organizational support; characteristics of the violence-related data elements; design of the ambulance run report form; and paramedic knowledge, attitudes, and behaviors regarding data collection.
DECEMBER 1999
34:6
ANNALS OF EMERGENCY MEDICINE
7 4 5
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
[Boergerhoff LA, Gerberich SG, Anderson A, Kochevar L, Waller L: Out-of-hospital violence injury surveillance: Quality of data collection. Ann Emerg Med December 1999;34:745-750.] INTRODUCTION
The need for improved data collection in injury prevention research has been recognized in national initiatives1,2 and by emergency medical services (EMS) leaders.3,4 Because paramedics and other emergency medical technicians (EMTs) respond to violence in their communities, they offer a unique opportunity to help identify the characteristics of the violent events and the persons affected. This is particularly important when victims of violence refuse ambulance transport and avoid subsequent medical care in the hospital emergency department. Out-of-hospital data collection has been problematic for reasons including the following: uncertainty about which data variables to collect; unavailability of desired data variables in the field; difficulties implementing and maintaining out-of-hospital data collection systems; the subjective nature of some out-of-hospital assessments; inadequate data collection instruments; and human factors such as disinterest, poor understanding, and lack of compliance with regard to data collection.5 Public health surveillance is the systematic and ongoing assessment of community health status through data collection, analysis, and interpretation. The purpose of the Prehospital Violence Injury Surveillance project was to examine the extent to which paramedics can adequately collect information about injuries, particularly intentional injuries, at emergency scenes. We present an assessment of paramedic interrater reliability and of the completeness and internal consistency of out-of-hospital data collection during a 3-month study period. M AT E R I A L S A N D M E T H O D S
Approval to conduct this study, under federal guidelines for the protection of human subjects in research, was granted by the authorized institutional review board. This study was conducted in a large medical transportation company in which approximately 140 paramedics respond to more than 44,000 emergency requests per year from an estimated resident population of 675,000 persons in a major Midwestern metropolitan area. Data were collected from ambulance run report forms that were filed on patients assessed by paramedics between November 1, 1996, and January 31, 1997.
7 4 6
A team of researchers reviewed and selected data elements from among a nationally recommended data set6 already included in the existing run report form that enabled a narrative description of the patient’s problem, medical history, physical examination, and treatments; patient demographics, vital signs, Glasgow Coma Scale scores, and billing information were also included. This form was then redesigned and expanded with additional variables thought to capture important characteristics of violence. These additions included the Injury Site/Type Matrix (ISTM), in which the paramedic coded the type of injury for a body site, and the Patient Assessment Profile (PAP), used to identify violence-related characteristics of the emergency event and the persons involved (Figure). Before implementing this effort, paramedics attended mandatory 4-hour violence-related training classes approximately 6 months before the study period to acquire assessment skills for violence-related injuries and screening of female patients for history and risk of domestic violence. The purpose of the study and the out-of-hospital data collection methods and protocols were also presented in this training. Paramedics used the redesigned run report form to document 3 classroom enactments and 5 written violence-related scenarios, which were later analyzed for interrater reliability. Paramedics began using the modified ambulance run report form in October and continued its use after completion of the study. To assist paramedics with data collection, a reference book was developed and placed in each ambulance before initiation of the study. Reminders, promoting data collection methods and protocols, were placed in the company’s newsletter before and during the study. Using the PAP, paramedics were to first categorize all patients into 1 of 3 problem types: injury, possible injury, or medical. Injuries were defined as signs of acute damage to body tissues, detectable by the paramedic using common field assessment methods. Possible, or suspected, injuries were those with a mechanism of injury and symptoms (eg, pain) yet lacking signs of injury. Medical problems included all other noninjury events. If the patient had an injury or possible injury, the paramedic was to collect subsequent data variables appropriate to the situation. Specifically, female subjects were to be asked about the history or risk of domestic violence, and those between the ages of 10 and 60 with injuries or possible injuries were to be asked about possible pregnancy. The paramedic was to indicate the source of injury, weapon used, or both, and determine from the most reliable information at the emergency scene whether the injury was intentional. If so, the event was considered to be violence-
ANNALS OF EMERGENCY MEDICINE
34:6 DECEMBER 1999
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
related and was subsequently assigned International Classification of Diseases, Ninth Revision E-codes 950-958 if suicide or self-inflicted injury and 960-968 if homicide or injury purposely inflicted by another person. 7 If another person caused the patient’s injuries, the
paramedic was to document that person’s relationship to the patient. Using decision rules, the study’s principal investigator abstracted each run report form to assess completeness and internal consistency and assigned to each report form
Figure.
The Injury Site/Type Matrix and the Patient Assessment Profile.
DECEMBER 1999
34:6
ANNALS OF EMERGENCY MEDICINE
7 4 7
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
1 of 16 possible 3-digit audit scores (Table). The audit score was a composite of 3 individual numeric codes (either “1” for error or “0” for no error) representing the combined responses to each of 3 questions about the run report form’s completeness and internal validity. For example, an audit score of 110 indicated that 2 types of errors occurred on the same report form: data were missing from the narrative (the first “1”), and the narrative contained information missing from the ISTM or PAP (the second “1”), while the ISTM and PAP did not contain inconsistencies between them or with the narrative (“0” in the third digit). To protect patient confidentiality, all personal identifiers (name, home address, home phone number, and Social Security number) were removed and a unique number was used to identify each ambulance run report form. The paramedics’ interrater reliability for each of the 3 enacted and 5 written scenarios from the training sessions was assessed using the κ statistic. κ Values greater than 0.75 indicate excellent interrater agreement, those between 0.4 and 0.75 indicate fair to good agreement, and values less than 0.4 indicate poor agreement.8 Second, the study’s principal investigator, a paramedic, rode with paramedics on 13 emergency calls during the study period and filled out independent run report forms that were later compared with the paramedics’ documentation. The completeness and reliability of the paramedics’ documentation was assessed through analysis of the audit codes assigned to each abstracted report form. Statistical analysis was performed using SAS (version 6.12, 1997, SAS Institute, Inc, Cary, NC) and Splus Table.
Components of the 3-digit audit code. Audit Code Digit First
Second
Third
7 4 8
Interpretation Was information missing from the narrative and from either the ISTM or PAP? 0=no (no missing data) 1=yes (missing data) Was information present in the narrative and missing from either the ISTM or PAP? 0=no (no missing data in ISTM or PAP) 1=yes (missing data in ISTM or PAP) Was any information inconsistent between narrative and ISTM and PAP? 0=no (no inconsistency) 1=yes (inconsistency in documentation between narrative and ISTM or PAP) 2=yes (new data in ISTM or PAP not in narrative) 3=yes (inconsistency and new data in ISTM or PAP not in narrative)
software (version Splus 3.4 for Unix, 1991, Statistical Sciences, Inc, Splus, Seattle, WA). R E S U LT S
Analysis of the data from the training, involving 3 enacted and 5 written classroom scenarios, was performed by obtaining the κ values for each scenario. The κ values ranged from 0.537 (95% confidence interval [CI] 0.533 to 0.541) to 0.937 (95% CI 0.933 to 0.941). The κ values indicated paramedic interrater reliability above 0.4 in all scenarios, and above 0.75, indicating excellent agreement, in 2 scenarios. The most frequent errors were missing data and inconsistent documentation of injuries between the narrative and the ISTM. Class discussion and evaluations revealed the paramedics’ dislike of additional data collection and skepticism of out-of-hospital data used to help monitor community violence. Many paramedics saw violence as a law enforcement problem and of no interest to EMS personnel beyond emergency patient care. Paramedics identified barriers to violence-related data collection including personal discomfort with pregnancy and violence screening questions, fear of legal liability for documenting injury intent and victim-perpetrator relationships, disapproval of a violence-related patient’s personal behaviors, and distrust of certain patient self-reported data. A comparison of the documentation between the paramedics and the principal investigator during the 13 ride-along observations was conducted. Among these 13 emergency events documented, there was disagreement on 10 (77%) of the report forms. Subsequently, an analysis was conducted of the 7,363 abstracted run report forms from independent emergency events that were filed by 133 paramedics. Of these patients, 4,020 (55%) were female, 3,319 (45%) patients were male, and 24 (0.3%) had no recorded gender. The patients’ ages ranged from younger than 1 year to 103 years (mean 49 years). Medical problems occurred in 4,262 (58%) patients, whereas 1,528 (21%) patients were injured, and 1,573 (21%) patients showed possible injuries. Of the 3,101 combined injuryrelated events, 2,685 (87%) were unintentional and 416 (13%) were considered intentional and violence-related, either self-inflicted or interpersonal. Among all 7,363 run report forms abstracted, 5,353 (73%) contained at least 1 documentation error in the data elements of interest. Of run report forms with errors, 99% contained errors of omission and 29% showed inconsistencies either between the narrative and the ISTM or PAP or between the ISTM and the PAP. More than half (56%) of all run report forms contained errors of more
ANNALS OF EMERGENCY MEDICINE
34:6 DECEMBER 1999
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
than one type or in multiple locations. Notably, documentation of violence screening of the 4,020 female patients was missing from more than 99% of the corresponding run report forms. The Glasgow Coma Scale score was missing on 28% of all run report forms. Complete vital signs (systolic blood pressure, pulse, and respiratory rate) were missing on 15% of all report forms, similar to vital sign omissions of approximately 17% previously reported. 9 Of 1,568 run report forms with documentation inconsistencies, 473 (30%) contained only inconsistencies between the narrative and the ISTM or PAP, 706 (45%) of the report forms showed new data collected in the ISTM or PAP that were not documented in the narrative, and 389 (25%) contained both inconsistencies and new data in the ISTM or PAP. Thus, 70% of all injuryrelated documentation inconsistencies reflected new data collected using the fill-in bubble format that were not otherwise documented in the narrative. Overall, injury-related run report forms were more likely to contain documentation errors than were those from medical events. Of the 3,101 run report forms from injury and possible injury events, 80% omitted data elements of interest from the narrative, 99% from the ISTM or PAP, and 47% from both the narrative and the ISTM and PAP. Notably, injury intent was missing from 72% of all injury-related run report forms, and the pregnancy status of females was missing on 66% of the corresponding report forms. DISCUSSION
This study of out-of-hospital data collection for violence injury surveillance has shown that paramedics are capable of good and sometimes excellent interrater agreement when documenting enacted and written class-room scenarios, using a structured reporting format such as the ISTM and the PAP. However, this study has also revealed widespread errors of omission and inconsistency in actual out-of-hospital data collection, particularly in data elements important to violence injury surveillance. Out-of-hospital injury surveillance relies on data collection previously recognized as both problematic and promising. Significant barriers to out-of-hospital data collection for violence injury surveillance were found and revealed as limitations in 3 areas: organizational support; data elements and collection methods; and paramedic knowledge, attitudes, and behaviors.
DECEMBER 1999
34:6
ANNALS OF EMERGENCY MEDICINE
Organizational support for this project was difficult to establish and maintain. The project was not integrated into new employee orientation, continuing education, or quality assurance programs, and paramedics interpreted this data collection as optional. A previous study indicated that strong organizational support has been shown to improve out-of-hospital data collection. 10 Reliance on the narrative format, associated with the original portion of the run report form, is problematic for quality data collection because documentation practices may vary widely among paramedics. For example, in the current study, paramedics used more than 600 different abbreviations, symbols, and acronyms to describe components of the patient’s history, assessment, and treatment. The check box or fill-in bubble format, specifically designed for this effort, may improve data clarity and abstracting efficiency and may actually result in collection of data otherwise omitted from the narrative description, particularly when numerous, discrete data elements are required. Finally, paramedic knowledge, attitudes, and behaviors regarding data collection in the relatively unsupervised out-of-hospital setting may significantly limit this opportunity to collect data that can contribute to a more comprehensive understanding and prevention of community violence. The paramedics’ reluctance to collect specific violence-related data elements, particularly regarding domestic abuse screening, female pregnancy status, and injury intent, suggests that these are new and sensitive areas of questioning. Yet, they are essential data elements in a prehospital violence injury surveillance system. We thank Jon Roesler, MS, senior epidemiologist, Injury and Violence Prevention Unit, Minnesota Department of Health, for his consultation in injury surveillance design.
REFERENCES 1. US Department of Health and Human Services: Healthy People 2000: National Health Promotion and Disease Prevention Objectives: Full Report, With Commentary. Washington, DC: Government Printing Office, 1990:242-244. 2. US Department of Health and Human Services: Position Papers from The Third National Injury Control Conference. Washington, DC: US Department of Health and Human Services, 1991:159-250. 3. US National Highway Traffic Safety Administration: Emergency Medical Services Agenda for the Future [DOT HS 808 142]. Washington, DC: Government Printing Office, 1996. (TD 8.2:EM 3/18) 4. Garrison HG, Foltin GL, Becker LR, et al: The role of emergency medical services in primary injury prevention. Ann Emerg Med 1997;30:4-91. 5. Spaite DW, Valenzuela TD, Meislin HW: Barriers to clinical evaluations of EMS systems: Problems associated with field data collection, in Fitch JJ (ed): Prehospital Care Administration: Issues, Readings, Cases. Carlsbad, CA: JEMS Publishing, 1995:533-543.
7 4 9
OUT-OF-HOSPITAL VIOLENCE INJURY SURVEILLANCE Boergerhoff et al
6. Uniform Pre-hospital Emergency Medical Services (EMS) Conference: Dean JM (ed): Final Report. Washington, DC: US Department of Transportation, 1994. 7. International Classification of Diseases, 9th Revision— Clinical Modification, ed 4. Los Angeles, CA: Practice Management Information Corporation, 1995:522-525. 8. Fleiss JL: Statistical Methods for Rates and Proportions, ed 2. New York: John Wiley & Sons, 1981:218. 9. Moss RL: Vital signs records omissions on prehospital patient encounter forms. Prehospital Disaster Med 1993;8:21-27. 10. Spaite DW, Hanion T, Criss ED, et al: Prehospital data entry compliance by paramedics after institution of a comprehensive EMS data collection tool. Ann Emerg Med 1990;19:12701273.
7 5 0
ANNALS OF EMERGENCY MEDICINE
34:6 DECEMBER 1999