- Email: [email protected]
Gauging Urgency
Correspondence Gauging Urgency
Robert W Derlet, MD
Gauging Urgency To the Editor.
Donna Kinser, MD
Reply James W Gill, MD, MPH False-Positive End-Tidal CO 2
Wolfgang H Male& . Katharina Koetter, MD
Reply Lance Montauk, JD, MD Andrew D Michads, MD Martitia Barsotti, MD Cadavers as Teachers
Randall Ellis, MD Reply
Dane M Chapman, MD, PhD Kenneth.] Rhee, MD John A Marx, MD Benjamin Honigman, MD Edward A Panac&, MD
Sally H Cavanaugh, PhD Copyright © by the American College of Emergency Physicians.
A study recently published in Annals that addressed the issue of consistency in making the determination of "urgency" is especially important in this era of managed care and its gatekeepers' judgment of the necessity of ED care [November 1996;28:474-479]. The study's authors conclude that health care professionals often disagree about urgency care in the ED. However, the study design throws the soundness of the conclusions into question. First, just 27 charts were reviewed by eight health care professionals to make a determination of urgency despite the statistical significance and "computerized random sampling." This is an extremely small number to analyze, and conclusions reached with the use of this sample must be guarded. Second, only limited data were used to make a decision: age, sex, medical history, medications, vital signs, and chief complaint. Therefore, even if conclusions were otherwise valid, their application would be limited. Another problem is the retrospective nature of the study. The authors do not disclose what actually happened to each of the patients reviewed. Information from these patients' ED
workups and outcomes would have provided more objective evidence of the actual need for urgent care. Finally, the criteria used to define urgency were not well defined; although the reviewers used the same general definition, no specific chief complaint guidelines were used. Agreement with regard to urgency category was only slight between the primary reviewer's retrospective assessment and the triage nurses' prospective assessments. This discrepancy raises several questions: Was the guiding definition of an urgent visit used by the triage nurses (major illness or injury in which a danger to the patient exists if the condition is not treated within 20 minutes to 2 hours) not identical to the one provided the retrospective reviewers? Was there a difference in the level of sophistication of the reviewer versus the triage nurse? Did the primary reviewer nurse and triage nurses have the same set of data? The triage nurses had the opportunity to directly view the patient and factor in subtle or obvious information from the physical examination. The authors suggested that the study simulates after-the-fact review by payors of visits to determine their urgency and that it may simulate steps taken leading to legal chal-
Guidelines for Letters Annalswelcomescorrespondence, including observations, opinions, corrections, very brief reports, and comments on published articles. Letters to the editor will not be accepted if they exceed three double-spaced pages, with a maximum of 10 references. Two double-spaced copies must be submitted; a computer disk is appreciated but not required. They should not contain abbreviations, Letters must be signed and include a postscript granting permission to publish. Financial associations or other possible conflicts of interest should always be disclosed. Letters discussing an Annalsarticle should be received within 6 weeks of the article's publication. Annalsacknowledges receipt of letters with a postcard, and correspondents are notified by postcard when e decision is made. Published letters will be edited and may be shortened. Unpublished letters will not be returned.
JULY 1997
30:1
ANNALS OF EMERGENCY MEDICINE
! t 5
CORRESPONDENCE
lenges of denial of emergency care for problems initially deemed nonurgent. It seems that in many such instances the decision would be based on review of the patient's entire course rather than just selected information from the triage record. However, on the positive side the analysis of information from the triage record to determine urgency is more similar to the decision analysis used by many health maintenance organizations when the ED calls to gain preauthorization for treatment. The HMO representatives often make decisions over the telephone on the basis of limited data. Again, because no correlation was found with the patient's ultimate diagnosis and care in this study, it is not possible to make a judgment as to whether the primary reviewer nurse or the triage nurses were more accurate in their assessment of urgency. However, the authors of other studies have tested systems involving hands-on evaluation by a triage nurse in making urgency determinations, and these systems have produced good results. The authors of one triage study presented outcome data from more than 5,000 patients who had been safely referred out of the ED after a screening examination by the triage nurse and the application of a set of criteria. 1 Recently authors from the University of Tennessee published the results of their experience with a nearly identical system. 2 These studies of urgency categorization and triage involving more comprehensive medical screening and physical examination should not be judged by Gill et al's retrospective chart study because they provide reasonable outcome data. Studies have shown that hands-on evaluation of the patient provides a good degree of reliability in making urgency determinations. In contrast, retrospective judgments using limited data, as indicated in this study, are fraught with wide variations in determinations of urgency and, consequently, with potential error.
Robert W Derlet, MD Donna Kinser, MD Division of EmergencyMedicine University of California, Davis, Medical Center Sacramento, CA I. Derlet RW, Kinser D, Ray L, et ah Prospective identification and triage of nonurgent patients out of an emergency department. Ann Emerg Med 1995;25:215-233. 2. Park LJ, Merigan KS, Fulbright V: Success of a triage-out program in a large public teaching hospital [abstract].
116
Society for Academic Emergency Medicine Annual Meeting, 1996:499.
In reply." This letter brings up some important points about the interpretation of our study. First, Dr Derlet and Dr Kinser note that in our study we were not able to determine the "accuracy" of ED triage assessments.We agree that our study does not examine whether triage assessments accurately predict diagnosis, outcome, or need for hospital admission. Nor did the study examine whether it is safe to refer patients out of the ED if they have been assessed as having a nonurgent condition. Our study was intended to measure the interrater reliability, not the predictive validity, of urgency assessments. Whether data on patient diagnosis, disposition, and outcome were reported is irrelevant to the purpose of the study. It is also noted that the criteria and data we used were not identical for the prospective and retrospective raters and that even among the retrospective raters who used identical criteria these criteria were somewhat subjective. Again, we agree. We intentionally allowed some subjectivity in the urgency ratings to replicate what really occurs in health care delivery. In the era of managed care, health professionals in the ED are often required to contact the patient's primary care provider, who decides whether to approve the ED visit on the basis of their assessment of urgency. In such cases, the ED provider and the primary care physician are unlikely to be using identical criteria; even if they are, these criteria certainly permit subjective judgments. In fact, the criteria used by these ED and primary care providers are probably less standardized than those used in our study. Therefore our study probably underestimates, rather than overestimates, the level of disagreement that would be found in such "real-world"
the 27 patients, we found a K-value of .38, with a 95% CI of .30 to .46. Although this interval might be wide, it is certainly not so wide as to "throw the soundness of the conclusions into question." Even if one takes the most optimistic estimate represented by the high end of this CI, the agreement between raters is only moderate and the conclusions of the study are unchanged. In summary, we found significant disagreement among health care professionals in their assessment of urgency in patients presenting to the ED. Another study of the interrater reliability of urgency assessments found very similar results.2 These findings certainly have limitations, including the fact that they are not an indication of the predictive value of urgency assessments or the safety of programs that refer nonurgent patients out of the ED. However, these findings suggest that problems can occur when a second health care professional is asked to reassess urgency for the purpose of approving an ED visit or approving payment for a visit that has already occurred. Given the high level of disagreement that we and others found, it is little wonder that approval or payment is often denied for problems that seemed obviously urgent to the ED provider who initially saw the patient.
James M Gill, MD, MPH Department of Family and Community Medicine Medical Center of Delaware Wilmington, DE 1. Fleiss J: Statistical methods for rates and proportions. In: Wiley, ed: The Measurement of Interrater Agreement, ed 2. New York: Wiley, 1981. 2. BriIlman JC, Doezema D, Tandberg D, et aI: Triage: Limitations in predicting need for emergent care and hospital admission. Ann Emerg Med 1996;27:493-500.
False-Positive End-Tidal
cases.
Finally, Dr Derlet and Dr Kinser make a statistical argument that the sample size was "an exceedingly small number to analyze." Although a sample of 27 patients seems small, there is little statistical basis for this argument. As in any study, the strength of the conclusions depends on the confidence interval (Cl) for the findings. Sample size is only one determinant of the range of the CI (in Fleiss' K-statistic used in our study,1 the width of the CI is determined by the sample size and the number of raters). Among the retrospective raters who assessed
CO 2 To the Editor: We read with interest the recent report of pharyngeal malposition of an endotracheal tube that went undetected by a colorimetric end-tidal CO2 detector [October 1996;28:458-459]. It should be noted, however, that this case and the case cited by the authors1 are not the only such cases published. Two other cases have been published in the anesthesia literature. 2,3 In these cases quantitative capnography showed a reg-
ANNALS OF EMERGENCY MEDICINE
30:1 JULY 1997
Robert W Derlet, MD
Gauging Urgency To the Editor.
Donna Kinser, MD
Reply James W Gill, MD, MPH False-Positive End-Tidal CO 2
Wolfgang H Male& . Katharina Koetter, MD
Reply Lance Montauk, JD, MD Andrew D Michads, MD Martitia Barsotti, MD Cadavers as Teachers
Randall Ellis, MD Reply
Dane M Chapman, MD, PhD Kenneth.] Rhee, MD John A Marx, MD Benjamin Honigman, MD Edward A Panac&, MD
Sally H Cavanaugh, PhD Copyright © by the American College of Emergency Physicians.
A study recently published in Annals that addressed the issue of consistency in making the determination of "urgency" is especially important in this era of managed care and its gatekeepers' judgment of the necessity of ED care [November 1996;28:474-479]. The study's authors conclude that health care professionals often disagree about urgency care in the ED. However, the study design throws the soundness of the conclusions into question. First, just 27 charts were reviewed by eight health care professionals to make a determination of urgency despite the statistical significance and "computerized random sampling." This is an extremely small number to analyze, and conclusions reached with the use of this sample must be guarded. Second, only limited data were used to make a decision: age, sex, medical history, medications, vital signs, and chief complaint. Therefore, even if conclusions were otherwise valid, their application would be limited. Another problem is the retrospective nature of the study. The authors do not disclose what actually happened to each of the patients reviewed. Information from these patients' ED
workups and outcomes would have provided more objective evidence of the actual need for urgent care. Finally, the criteria used to define urgency were not well defined; although the reviewers used the same general definition, no specific chief complaint guidelines were used. Agreement with regard to urgency category was only slight between the primary reviewer's retrospective assessment and the triage nurses' prospective assessments. This discrepancy raises several questions: Was the guiding definition of an urgent visit used by the triage nurses (major illness or injury in which a danger to the patient exists if the condition is not treated within 20 minutes to 2 hours) not identical to the one provided the retrospective reviewers? Was there a difference in the level of sophistication of the reviewer versus the triage nurse? Did the primary reviewer nurse and triage nurses have the same set of data? The triage nurses had the opportunity to directly view the patient and factor in subtle or obvious information from the physical examination. The authors suggested that the study simulates after-the-fact review by payors of visits to determine their urgency and that it may simulate steps taken leading to legal chal-
Guidelines for Letters Annalswelcomescorrespondence, including observations, opinions, corrections, very brief reports, and comments on published articles. Letters to the editor will not be accepted if they exceed three double-spaced pages, with a maximum of 10 references. Two double-spaced copies must be submitted; a computer disk is appreciated but not required. They should not contain abbreviations, Letters must be signed and include a postscript granting permission to publish. Financial associations or other possible conflicts of interest should always be disclosed. Letters discussing an Annalsarticle should be received within 6 weeks of the article's publication. Annalsacknowledges receipt of letters with a postcard, and correspondents are notified by postcard when e decision is made. Published letters will be edited and may be shortened. Unpublished letters will not be returned.
JULY 1997
30:1
ANNALS OF EMERGENCY MEDICINE
! t 5
CORRESPONDENCE
lenges of denial of emergency care for problems initially deemed nonurgent. It seems that in many such instances the decision would be based on review of the patient's entire course rather than just selected information from the triage record. However, on the positive side the analysis of information from the triage record to determine urgency is more similar to the decision analysis used by many health maintenance organizations when the ED calls to gain preauthorization for treatment. The HMO representatives often make decisions over the telephone on the basis of limited data. Again, because no correlation was found with the patient's ultimate diagnosis and care in this study, it is not possible to make a judgment as to whether the primary reviewer nurse or the triage nurses were more accurate in their assessment of urgency. However, the authors of other studies have tested systems involving hands-on evaluation by a triage nurse in making urgency determinations, and these systems have produced good results. The authors of one triage study presented outcome data from more than 5,000 patients who had been safely referred out of the ED after a screening examination by the triage nurse and the application of a set of criteria. 1 Recently authors from the University of Tennessee published the results of their experience with a nearly identical system. 2 These studies of urgency categorization and triage involving more comprehensive medical screening and physical examination should not be judged by Gill et al's retrospective chart study because they provide reasonable outcome data. Studies have shown that hands-on evaluation of the patient provides a good degree of reliability in making urgency determinations. In contrast, retrospective judgments using limited data, as indicated in this study, are fraught with wide variations in determinations of urgency and, consequently, with potential error.
Robert W Derlet, MD Donna Kinser, MD Division of EmergencyMedicine University of California, Davis, Medical Center Sacramento, CA I. Derlet RW, Kinser D, Ray L, et ah Prospective identification and triage of nonurgent patients out of an emergency department. Ann Emerg Med 1995;25:215-233. 2. Park LJ, Merigan KS, Fulbright V: Success of a triage-out program in a large public teaching hospital [abstract].
116
Society for Academic Emergency Medicine Annual Meeting, 1996:499.
In reply." This letter brings up some important points about the interpretation of our study. First, Dr Derlet and Dr Kinser note that in our study we were not able to determine the "accuracy" of ED triage assessments.We agree that our study does not examine whether triage assessments accurately predict diagnosis, outcome, or need for hospital admission. Nor did the study examine whether it is safe to refer patients out of the ED if they have been assessed as having a nonurgent condition. Our study was intended to measure the interrater reliability, not the predictive validity, of urgency assessments. Whether data on patient diagnosis, disposition, and outcome were reported is irrelevant to the purpose of the study. It is also noted that the criteria and data we used were not identical for the prospective and retrospective raters and that even among the retrospective raters who used identical criteria these criteria were somewhat subjective. Again, we agree. We intentionally allowed some subjectivity in the urgency ratings to replicate what really occurs in health care delivery. In the era of managed care, health professionals in the ED are often required to contact the patient's primary care provider, who decides whether to approve the ED visit on the basis of their assessment of urgency. In such cases, the ED provider and the primary care physician are unlikely to be using identical criteria; even if they are, these criteria certainly permit subjective judgments. In fact, the criteria used by these ED and primary care providers are probably less standardized than those used in our study. Therefore our study probably underestimates, rather than overestimates, the level of disagreement that would be found in such "real-world"
the 27 patients, we found a K-value of .38, with a 95% CI of .30 to .46. Although this interval might be wide, it is certainly not so wide as to "throw the soundness of the conclusions into question." Even if one takes the most optimistic estimate represented by the high end of this CI, the agreement between raters is only moderate and the conclusions of the study are unchanged. In summary, we found significant disagreement among health care professionals in their assessment of urgency in patients presenting to the ED. Another study of the interrater reliability of urgency assessments found very similar results.2 These findings certainly have limitations, including the fact that they are not an indication of the predictive value of urgency assessments or the safety of programs that refer nonurgent patients out of the ED. However, these findings suggest that problems can occur when a second health care professional is asked to reassess urgency for the purpose of approving an ED visit or approving payment for a visit that has already occurred. Given the high level of disagreement that we and others found, it is little wonder that approval or payment is often denied for problems that seemed obviously urgent to the ED provider who initially saw the patient.
James M Gill, MD, MPH Department of Family and Community Medicine Medical Center of Delaware Wilmington, DE 1. Fleiss J: Statistical methods for rates and proportions. In: Wiley, ed: The Measurement of Interrater Agreement, ed 2. New York: Wiley, 1981. 2. BriIlman JC, Doezema D, Tandberg D, et aI: Triage: Limitations in predicting need for emergent care and hospital admission. Ann Emerg Med 1996;27:493-500.
False-Positive End-Tidal
cases.
Finally, Dr Derlet and Dr Kinser make a statistical argument that the sample size was "an exceedingly small number to analyze." Although a sample of 27 patients seems small, there is little statistical basis for this argument. As in any study, the strength of the conclusions depends on the confidence interval (Cl) for the findings. Sample size is only one determinant of the range of the CI (in Fleiss' K-statistic used in our study,1 the width of the CI is determined by the sample size and the number of raters). Among the retrospective raters who assessed
CO 2 To the Editor: We read with interest the recent report of pharyngeal malposition of an endotracheal tube that went undetected by a colorimetric end-tidal CO2 detector [October 1996;28:458-459]. It should be noted, however, that this case and the case cited by the authors1 are not the only such cases published. Two other cases have been published in the anesthesia literature. 2,3 In these cases quantitative capnography showed a reg-
ANNALS OF EMERGENCY MEDICINE
30:1 JULY 1997