- Email: [email protected]
Validity of Selected Patient Safety Indicators: Opportunities and Concerns
Validity of Selected Patient Safety Indicators: Opportunities and Concerns Haytham MA Kaafarani, MD, MPH, Ann M Borzecki, MD, MPH, Kamal MF Itani, MD, FACS, Susan Loveland, MAT, Hillary J Mull, MPP, Kathleen Hickson, RN, MN, Sally MacDonald, RNC, Marlena Shin, JD, MPH, Amy K Rosen, PhD The Agency for Healthcare Research and Quality (AHRQ) recently designed the Patient Safety Indicators (PSIs) to detect potential safety-related adverse events. The National Quality Forum has endorsed several of these ICD-9-CM-based indicators as quality-of-care measures. We examined the positive predictive value (PPV) of 3 surgical PSIs: postoperative pulmonary embolus and deep vein thrombosis (pPE/DVT), iatrogenic pneumothorax (iPTX), and accidental puncture and laceration (APL). STUDY DESIGN: We applied the AHRQ PSI software (v.3.1a) to fiscal year 2003 to 2007 Veterans Health Administration (VA) administrative data to identify (flag) patients suspected of having a pPE/DVT, iPTX, or APL. Two trained nurse abstractors reviewed a sample of 336 flagged medical records (112 records per PSI) using a standardized instrument. Inter-rater reliability was assessed. RESULTS: Of 2,343,088 admissions, 6,080 were flagged for pPE/DVT (0.26%), 1,402 for iPTX (0.06%), and 7,203 for APL (0.31%). For pPE/DVT, the PPV was 43% (95% CI, 34% to 53%); 21% of cases had inaccurate coding (eg, arterial not venous thrombosis); and 36% featured thromboembolism present on admission or preoperatively. For iPTX, the PPV was 73% (95% CI, 64% to 81%); 18% had inaccurate coding (eg, spontaneous pneumothorax), and 9% were pneumothoraces present on admission. For APL, the PPV was 85% (95% CI, 77% to 91%); 10% of cases had coding inaccuracies and 5% indicated injuries present on admission. However, 27% of true APLs were minor injuries requiring no surgical repair (eg, small serosal bowel tear). Inter-rater reliability was ⬎90% for all 3 PSIs. CONCLUSIONS: Until coding revisions are implemented, these PSIs, especially pPE/DVT, should be used primarily for screening and case-finding. Their utility for public reporting and pay-for-performance needs to be reassessed. (J Am Coll Surg 2011;212:924–934. © 2011 by the American College of Surgeons) BACKGROUND:
health care priority.1,2 The number of studies aimed at measuring adverse events, safety-related events, errors, or patient harm has increased exponentially in the last few years. Medical record review promises richer clinical detail in identifying adverse events, but is resource-intensive and expensive.3 Traditional mortality and morbidity conferences show less than 25% sensitivity in detecting adverse events when compared with the large chart-based quality measurement initiatives such as the National Surgical Quality Improvement Program (NSQIP).4,5 Recently, in an effort to improve and increase detection of potentially preventable safety events in acute-care hospitals, the Agency for Healthcare Research and Quality (AHRQ) designed a set of evidence-based ICD-9-CMbased algorithms called Patient Safety Indicators (PSIs). The PSIs represent a significant advance in the field of patient safety because they use readily available hospital discharge data that can be risk adjusted to screen for safetyrelated events in the inpatient setting.6 The PSIs were ini-
Since publication of the Institute of Medicine’s 2 landmark reports, “To Err is Human” and “Crossing the Quality Chasm” in 2000, patient safety has become a national Disclosure Information: Nothing to disclose. Abstract presented American College of Surgeons 95th Annual Clinical Congress, Surgical Forum, Chicago, IL, October 2009. Received April 19, 2010; Revised July 8, 2010; Accepted July 8, 2010. From the Department of Surgery, Tufts Medical Center, Tufts University School of Medicine (Kaafarani), VA Boston Healthcare System and the Center for Health Quality, Outcomes and Economic Research (Kaafarani), Boston University School of Public Health (Borzecki, Mull, Rosen), Boston University School of Medicine (Borzecki, Itani), Harvard Medical School (Itani), and the Center for Organization, Leadership and Management Research, VA Boston Healthcare System (Loveland, Mull, Shin, Rosen), Boston, MA; the Center for Health Quality, Outcomes and Economic Research, Bedford VAMC, Bedford, MA (Borzecki, Hickson, MacDonald); and the Department of Surgery, VA Boston Healthcare System, West Roxbury, MA (Itani, Hickson). Correspondence address: Amy K Rosen, PhD, Center for Organization, Leadership and Management Research (COLMR), VA Boston Healthcare System, 150 S Huntington Ave, Boston MA 02130. email: [email protected] or [email protected]
© 2011 by the American College of Surgeons Published by Elsevier Inc.
924
ISSN 1072-7515/11/$36.00 doi:10.1016/j.jamcollsurg.2010.07.007
Vol. 212, No. 6, June 2011
Kaafarani et al
Abbreviations and Acronyms
AHRQ APL CMS iPTX IRR pPE/DVT
⫽ ⫽ ⫽ ⫽ ⫽ ⫽
PPV PSI VA VTE
⫽ ⫽ ⫽ ⫽
Agency for Healthcare Research and Quality accidental puncture and laceration Center for Medicare and Medicaid Services iatrogenic pneumothorax inter-rater reliability postoperative pulmonary embolus and deep vein thrombosis positive predictive value Patient Safety Indicator Veterans Health Administration venous thromboembolism
tially intended as screening tools to identify safety-related events or as case-finding tools for internal quality improvement purposes. However, in the last several years, they have been used by multiple organizations for hospital profiling and pay-for-performance purposes.7,8 Eight of the PSIs have been endorsed by the National Quality Forum (NQF) as hospital performance measures, and 4 (plus 1 composite measure) have been adopted by the Centers for Medicare and Medicaid Services (CMS) for hospital comparisons of quality and safety and financial reimbursement.9,10 With the increased use of PSIs for quality assessment, public reporting, hospital profiling, and reimbursement,8,11 many clinicians, policymakers, and researchers have raised concerns over the validity of PSIs, given that the PSI algorithms are based on administrative data that are well known for their variability and inconsistency in the coding of diagnoses and procedures. Some of these coding issues include ambiguity in ICD-9-CM coding guidelines, variation in coding practices across different hospitals, and the codes’ inability to differentiate between events that happened de novo versus those that were present on admission.12,13 Although the current literature suggests that PSIs examined to date have moderate to high sensitivities and specificities, their measured performance has been found to depend on the nature of the specific PSI, the nature of the adverse events it targets, and the method of validation used to test its performance. In these multiple validation attempts, the calculated positive predictive values (PPVs) of PSIs against chart abstraction as the “gold standard,” ranged between 44% and 91%.14-20 Although the criterion validity of several PSIs against the gold standard has been examined in the private sector, little is known about whether these results are reliable in other health care settings, such as the Veterans Health Administration (VA). In this study, we examined the positive PPV of 3 surgical PSIs against medical record review: postoperative pulmonary embolus and deep vein thrombosis (pPE/ DVT), iatrogenic pneumothorax (iPTX), and accidental
Validity of Selected Patient Safety Indicators
925
puncture and laceration (APL). These 3 PSIs have all been endorsed by the National Quality Forum as quality measures; APL and iPTX have been adopted by CMS.
METHODS We applied the AHRQ PSI software (v.3.1a) to VA fiscal year 2003 to 2007 (October 1, 2002 to September 30, 2007) administrative data from a sample of 28 VA hospitals to identify (flag) patients suspected of having a pPE/DVT, iPTX, or APL. To determine rates of true and false positives for each PSI, trained nurses conducted a retrospective chart review of 336 flagged charts (112 charts per PSI) using standardized chart abstraction tools and guidelines developed by the AHRQ and modified for the VA’s electronic medical record. Inter-rater reliability testing of the 2 nurse abstractors was also performed. The required Institutional Review Board (IRB) approvals from the Bedford VA Medical Center and the VA Boston Healthcare System were obtained to conduct this study. PSI overview and definitions
PSIs are calculated as rates: the numerator consists of all discharges with diagnoses, procedures, and/or other attributes indicating the occurrence of the particular adverse event; the denominator includes all discharges at risk for the adverse event, with certain exclusions. All denominators include patients aged 18 and older and exclude patients with Major Diagnostic Category (MDC) 14 (pregnancy, childbirth, and puerperium).21 Postoperative PE/DVT detects the occurrence of postoperative venous thromboembolism (VTE) by identifying discharges that have any of the ICD-9-CM codes for PE or DVT in a secondary diagnosis field. The denominator includes all surgical discharges as defined by specific Diagnosis-Related Groups (DRGs) and an ICD-9-CM code for an operating room procedure, narrowing the denominator to those cases most likely to be preventable. It excludes all patients with pre-existing (principal diagnosis or secondary diagnosis present on admission, if known) PE or DVT and patients who underwent a procedure to interrupt the vena cava either as the only operating room procedure during admission or before or on the same day of the index operating room procedure. The occurrence of iPTX is targeted by identifying discharges with the ICD-9-CM code 512.1 in any secondary diagnosis field. The denominator includes all surgical and medical discharges of patients with specific DRGs. It excludes patients with the code 512.1 in a primary diagnosis field; a code for diaphragmatic surgery repair; a cardiac surgery DRG; a thoracic surgery or lung or pleural biopsy code; and a diagnosis code for chest trauma or pleural ef-
926
Kaafarani et al
Validity of Selected Patient Safety Indicators
J Am Coll Surg
Figure 1. Hospital sampling strategy. PSI, Patient Safety Indicator.
fusion. Iatrogenic pneumothoraces caused by procedures performed within 72 hours before admission are included. Similar to pPE/DVT, the denominator was purposely restrictive in order to capture the most “preventable” cases. APL captures the occurrence of accidental cut, puncture, perforation, or hemorrhage during medical care (ICD9-CM codes E870.0 through E870.9) or accidental puncture or laceration during a procedure (ICD-9-CM code 998.2) in a secondary diagnosis field. The denominator includes all surgical and medical discharges excluding patients whose APL was present on admission or in a primary diagnosis field.
the expected and observed numerators in any of the PSIs were excluded, yielding a final sample of 79 hospitals. Within each tier of hospitals, facilities were ranked using the AHRQ PSI composite rate.21 The top 3 and bottom 3 hospitals were included in the hospital sample. We then randomly selected from the remaining hospitals within each stratum to obtain a sample of 28 hospitals. To assure balanced geographic representation, 3 hospitals were replaced by the next hospital in rank, for a final sample of 28 hospitals representing diverse geographic regions of the US. The hospital selection process is shown in Figure 1. Case selection
Hospital selection
Our initial hospital sample included 158 acute-care VA hospitals. To obtain a manageable number of hospitals for chart review, and to minimize variation in coding across hospitals, we selected a sample of hospitals from the 158 that represented a broad spectrum of PSI rates. We grouped the 158 hospitals into 3 tiers based on their observed and expected rates of PSIs, exclusive of PSI 5 (foreign body left during procedure) and PSI 8 (postoperative hip fracture), both of which had low incidence rates across most hospitals. The expected number of PSI events of a specific facility was calculated as the national VA PSI rate multiplied by the PSI denominator of that specific facility. The first group of hospitals included facilities that had a numerator of at least 4 safety-related events in both the expected and observed numerators of each PSI. The second group had at least 2 safety-related events in the expected and observed numerators of each PSI. The third group had at least 1 safetyrelated event in the expected and observed numerators of each PSI. Hospitals with less than 1 safety-related event in
Four flagged medical records per PSI (pPE/DVT, iPTX, APL) were randomly selected from each of the 28 hospitals for a total of 336 medical records. Based on previously reported PPV estimates, 112 cases per PSI were selected to ensure reasonably narrow PPV confidence intervals (range 10% to 20%). When a certain facility had less than 4 flagged records for a specific PSI, flagged records were reviewed from the next facility within the stratum, for a maximum of 8 records total per hospital. Medical record abstraction
Two trained nurses used standardized abstraction instruments to review medical records for occurrence of a safetyrelated event (pPE/DVT, iPTX, or APL); demographics, comorbidities, and risk factors of the patient population; clinical circumstances surrounding the safety-related event; and patient outcomes after the event. Nurse training included several sessions discussing the rationale behind each PSI, the likely sources of information needed from the electronic medical record, and a systematic chronology for
Vol. 212, No. 6, June 2011
Kaafarani et al
chart abstraction. When the nurse abstractors were uncertain about any item, the medical record was referred to the research team physicians (HK, AB, KI) for resolution.
Validity of Selected Patient Safety Indicators
927
RESULTS Of 2,343,088 admissions, 6,080 were flagged for pPE/ DVT (0.26%), 1,402 for iPTX (0.06%), and 7,203 for APL (0.31%).
Inter-rater reliability
After completion of nurse training, we initiated inter-rater reliability (IRR) testing, as recommended in the literature, to obtain a standardized and reliable method of abstraction.22 At least 10% of the medical records were reviewed by both nurses, and IRR was measured as the percentage of agreement on select key questions of each abstraction tool, such as ascertainment of the safety-related event. Records were abstracted in groups of 5 until ⬎90% agreement was obtained; this typically occurred after the first or second round of IRR, and then nurses were instructed to proceed by abstracting separate charts. After each round of IRR (including the round achieving ⬎90%), a discussion of disagreements took place in the presence of the clinical physicians (HK, AB, KI) with resulting instrument revisions and/or guideline clarifications as appropriate. An additional round of IRR was performed on 5 records toward the end of the abstraction process to check for potential abstractor drift. Positive predictive value
For each of the 3 PSIs, we calculated PPV as the rate of true positives divided by the number of medical records reviewed and derived 95% confidence intervals for that estimate.
Postoperative pulmonary embolus or deep vein thrombosis Positive predictive value
Of 112 cases, 48 were true events of postoperative PE or DVT, yielding a PPV of 43% (95% CI 34% to 53%). IRR between chart abstractors was measured at 94%. True positive analysis
As shown in Table 1, the patient population was entirely male, with a mean age of 70 years. Seventy-one percent were white, non-Hispanic patients. The mean number of comorbidities per patient was 1.6: 17% of the patients were diabetic, 17% had chronic pulmonary disease, 13% had a malignancy, and 8% had congestive heart failure. Twentytwo of the 48 true pPE/DVT patients had only a DVT diagnosis (46%), 19 patients had only a PE diagnosis (40%), and 7 patients had both a DVT and a PE diagnosed (15%). Of the total of 29 DVTs, 23 involved the lower extremity (79%), and only 4 involved the upper extremities (14%). Orthopaedic and abdominal procedures accounted for more than half of the index procedures preceding the VTE events (38% and 21%, respectively); 67% of the index procedures were elective in nature. The median length of stay for these admissions was 22 days. All-cause inhospital mortality was 19%.
True positive analysis
For patients with a confirmed pPE/DVT, iPTX, or APL, we performed descriptive analyses of multiple continuous and categorical variables including demographics (age, gender, and race or ethnicity), comorbidities, relevant risk factors, and the nature of the surgical procedure and outcomes. For example, for pPE/DVT, the location of the DVT and the index procedure preceding the PE or DVT were identified. For iPTX, we examined the cause of the pneumothorax (eg, subclavian or internal jugular central line insertion, ventilator-associated barotrauma) and the training level of the staff involved in the procedure (attending vs trainee). For APL, we examined the nature of the surgical procedure, the type of puncture or laceration (eg, bowel injury, durotomy, and splenic injury), and the training level of the staff primarily involved in the procedure. All statistical analyses were performed using SAS version 9.1. False positive analysis
All false positive cases underwent further detailed review to better understand why they were incorrectly flagged by the PSI algorithms.
False positive analysis
Of a total of 64 false positive cases, 16 (25%) patients had a PE and/or DVT diagnosis present on admission (diagnosed 6 months or less before admission), 10 patients (16%) had a remote history of PE/DVT (diagnosed more than 6 months before admission), and 14 patients (22%) were diagnosed after admission but before the index procedure. Coding-related inaccuracies accounted for the remaining 24 false positives (38%) (Fig. 2). These included cases of arterial (not venous) thrombosis, superficial (not deep) vein thrombosis or thrombophlebitis, and cases in which a postoperative PE and DVT workup was negative or the etiology for patient postoperative mortality was uncertain (ie, “rule out” PE). Ten of the cases (9%) were considered false positives for diverse reasons (the miscellaneous category). For example, one of the false positive cases had a discharge summary stating: “patient with history and PE compatible with acute appendicitis.” It seems likely that, in this instance, the coder mistook the abbreviation “PE” to stand for pulmonary embolus rather than physical examination.
928
Kaafarani et al
Validity of Selected Patient Safety Indicators
Table 1. Analysis of True Positives for Postoperative Pulmonary Embolus or Deep Vein Thrombosis Variable
n Demographics Age, y, mean (SD) Gender, male, n (%) Race/ethnicity, n (%) White, non-Hispanic African American, non-Hispanic Hispanic Other/missing Body mass index, kg/m2, mean (SD) Comorbidities No. of comorbidities, mean (SD) No. of comorbidities, median (IQR) Specific comorbidities, n (%) Diabetes mellitus Chronic pulmonary disease Congestive heart failure Renal failure Solid malignancy without metastasis Metastatic malignancy Lymphoma Obesity Paralysis Other neurologic disorders Additional risk factors, n (%) Hypercoagulable state Baseline inability to ambulate History of spinal cord injury History of recent trauma Central venous catheter insertion Femoral Subclavian Internal jugular Peripherally inserted central line Ventilator dependence Chemotherapy Use of potentially procoagulant medication Transfusion of blood or blood products Index procedures, n (%) Elective procedure Procedure type Orthopaedic Abdominal Urologic (including nephrectomy) Thoracic/pulmonary Cardiac Neurosurgical Vascular
Data
48 70 (11) 48 (100) 34 (71) 5 (10) 3 (6) 6 (13) 27.1 (4.7) 1.6 (1.3) 1 (1.5 [1–2.5]) 8 (17) 8 (17) 4 (8) 3 (6) 3 (6) 2 (4) 1 (2) 1 (2) 1 (2) 2 (4) 2 (4) 5 (10) 1 (2) 0 (0) 1 (2) 5 (10) 7 (15) 13 (27) 8 (17) 2 (4) 0 (0) 0 (0)
J Am Coll Surg
Table 1. Continued Variable
Type of venous thromboembolism, n (%) Pulmonary embolus only Deep vein thrombosis only Both pulmonary embolus and deep vein thrombosis Location of deep vein thrombosis, when applicable, n⫽ 29, n (%) Lower extremity Upper extremity Internal jugular Undocumented location Outcomes Length of stay, d Mean (SD) Median (range) All-cause mortality during admission, n (%)
Data
19 (40) 22 (46) 7 (15)
23 (79) 4 (14) 1 (3) 1 (3)
35 (32) 22 (1–127) 9 (19)
Numbers might not add to totals due to rounding. IQR, interquartile range.
Iatrogenic pneumothorax Positive predictive value
Out of 112 cases reviewed, 82 were true iatrogenic pneumothoraces, yielding a PPV of 73% (95% CI, 64% to 81%]. IRR between the chart abstractors was measured at 94%. True positive analysis
As shown in Table 2, the population was 94% male and 62% white non-Hispanic; mean age was 68 years, and the mean body mass index was 24.5 kg/m2. Hypertension, airway lung disease (eg, asthma, COPD), and diabetes mel-
32 (67) 18 (38) 10 (21) 6 (13) 6 (13) 3 (6) 3 (6) 2 (4) (continued)
Figure 2. Positive predictive value and analysis of false positives of postoperative pulmonary embolus or deep vein thrombosis (pPE/ DVT). Numbers might not add to totals due to rounding. The percentages reported in the figure refer to the percentage of the total number of cases; those reported in the text of the manuscript refer to the percentage of the false positive cases only.
Vol. 212, No. 6, June 2011
Kaafarani et al
Table 2. Analysis of True Positives for Iatrogenic Pneumothorax Variable
n Demographics Age, y, mean (SD) Gender, male, n (%) Race/ethnicity, n (%) White, non-Hispanic African American, non-Hispanic Hispanic Other/missing Body mass index, kg/m2, mean (SD) Comorbidities No. of comorbidities, mean (SD) No. of comorbidities, median (IQR) Specific comorbidities, n (%) Hypertension Diabetes mellitus Chronic pulmonary disease Congestive heart failure Weight loss Alcohol abuse Liver disease Metastatic malignancy Peripheral vascular disease Pulmonary-related risk factors, n (%) Airway disease (eg, COPD, asthma) Respiratory tract malignancy Infectious lung disease (eg, pneumonia) Pleural effusion Interstitial lung disease (eg, lung fibrosis) Other respiratory disease Description of the iatrogenic pneumothorax Causes of the pneumothorax, n (%)* Central venous catheter insertion Transthoracic needle aspiration (or biopsy) Cardiac pacemaker placement Implantable defibrillator insertion Mechanical ventilation (barotrauma) Liver biopsy/liver lesion radiofrequency ablation Cardiopulmonary resuscitation Miscellaneous procedures near chest/neck Level of training of person performing procedure, n (%), n ⫽ 85 Attending Physician-in-training Unknown Outcomes Length of stay, d Mean (SD) Median (range) All-cause mortality during admission, n (%) Numbers might not add to totals due to rounding. *Three patients had 2 potential causative procedures each. IQR, interquartile range.
Data
82 68 (14) 77 (94) 51 (62) 12 (15) 4 (5) 15 (18) 24.5 (6.5) 1.7 (1.3) 2 (1 [1–2]) 31 (38) 19 (23) 19 (23) 12 (15) 6 (7) 4 (5) 2 (2) 2 (2) 2 (2) 24 (29) 9 (11) 8 (10) 7 (8) 3 (4) 5 (6)
31 (38) 15 (18) 11 (13) 4 (5) 3 (4) 3 (4) 1 (1) 14 (17)
28 (33) 38 (45) 19 (22)
18 (21) 12 (1–140) 15 (18)
Validity of Selected Patient Safety Indicators
929
litus were the most prevalent risk factors in the patient population (38%, 29%, and 23%, respectively). Of 82 cases of true iatrogenic pneumothoraces, 31 were caused by central venous catheter insertions (38%), 15 by transthoracic needle aspirations or biopsies (18%), 15 by cardiac pacemaker or defibrillator placement (18%), 3 by liver biopsies or liver lesion radiofrequency ablation (4%), 3 by mechanical ventilation (4%), and 1 by cardiopulmonary resuscitation (1%). Several miscellaneous procedures (eg, brachial, axillary or intercostal nerve blocks, pericardiocentesis) accounted for 14 pneumothoraces (17%). Forty-five percent of the pneumothoraces occurred during procedures performed by physicians in training; 33% occurred during procedures performed by attending-level physicians alone. The median length of hospital stay for these patients was 12 days, and the all-cause in-hospital mortality was 18%. False positive analysis
There were 30 false positive cases, including 10 patients with an old history of pneumothorax or pneumothorax present on admission (more than 72 hours before admission), and 20 cases of inaccurate coding (Fig 3). The latter included cases in which the pneumothorax was spontaneous but not iatrogenic; cases in which lung consolidation or collapse due to pneumonia or empyema were mistaken for iatrogenic pneumothoraces; and cases in which the procedure was known a priori to breach the pleural cavity, such as a thoracic procedure or a lung biopsy. Accidental puncture or laceration Positive predictive value
Ninety-five of 112 cases represented “true” cases of accidental punctures or lacerations, yielding a PPV of 85% (95% CI, 77% to 91%). IRR between the chart abstractors was measured at 97%. True positive analysis
As illustrated in Table 3, the population was 96% male and 63% white non-Hispanic; mean age was 67 years, and mean body mass index was 27.8 kg/m2. The mean number of comorbidities was 1.5, with the most common comorbidities being hypertension (57%), diabetes mellitus (22%), and chronic pulmonary disease (12%). Seventynine of the 95 true iatrogenic events involved the chest or the abdomen (83%); 75 of 95 occurred in the operating room (79%). Injury to vascular structures (inadvertent injury to large arteries or veins), bowel (eg, iatrogenic enterotomy), abdominal organs (eg, splenic injury in a colectomy), genitourinary organs (eg, ureteral injury), and spinal dura (eg, durotomy) accounted for 81% of the punctures or lacerations (23%, 17%, 15%, 13%, and 11%,
930
Kaafarani et al
Validity of Selected Patient Safety Indicators
Figure 3. Positive predictive value and analysis of false positives of iatrogenic pneumothorax (iPTX). Numbers might not add to totals due to rounding. The percentages reported in the figure refer to the percentage of the total number of cases; those reported in the text of the manuscript refer to the percentage of the false positive cases only. POA, present on admission.
respectively). Simple serosal bowel tears accounted for 6% of the injuries. When management of these injuries was examined, 26% of them required no surgical management at all (eg, a small serosal bowel tear that was not repaired or a small spinal durotomy that was managed using gelfoam only). Forty-four percent of these injuries occurred during procedures performed by physicians in training; 37% occurred during procedures performed by attending-level physicians alone. In 25% of cases, surgeons reported presence of adhesions or scar tissue; an additional 5% were associated with abnormal anatomy. The median length of stay for patients who sustained these injuries was 8 days. The all-cause in-hospital mortality of these patients was 5%. False positive analysis
Out of a total of 17 false positive cases, 6 were patients with punctures or lacerations that were present on admission (35%) and 11 were cases of inaccurate coding diagnosis (65%) (Fig. 4). Half of the cases in the latter category were due to the puncture or laceration being nonaccidental (eg, ampulla of Vater “slit” during an endoscopic retrograde cholangiopancreatography to release impacted stones; spontaneous nonprocedure sigmoid colon perforation). The other half involved miscellaneous conditions (eg, suspicion of an intubation-related pharyngeal injury that was later ruled out), including cases in which the reason for coding as an accidental puncture or laceration remained unclear to the chart abstractors.
DISCUSSION This is the first study to examine the validity of pPE/DVT, iPTX, and APL in the VA. We found that the PPVs of these
J Am Coll Surg
3 PSIs varied considerably. Postoperative PE/DVT had a low PPV (43%), iPTX had a moderate PPV (74%), and APL had a moderate-to-high PPV (85%). However, 26% of the accidental punctures or lacerations detected by the APL algorithm were minor injuries that did not require any surgical repair or intervention (eg, serosal bowel tears or dural tears that heal with simple observation or with the use of sealants). Our PPV findings are consistent with those from private sector studies.14-20,23 Two recent studies estimated the PPV of pPE/DVT to be between 44% and 55%.14 A separate study estimated the PPV of APL to be around 91%, but similarly noted that 24% of the injuries were “inconsequential,” “expected to heal without repair,” and for which “the risk may have been acceptable relative to the goals of the procedure.”16 Another recent study estimated the PPV of iPTX to be around 78%, similar to our findings, with 44% of the pneumothoraces resulting from central venous line insertions.23 Policy implications
The PSIs we studied show promise as screening tools that can be used to detect patient safety events related to postoperative VTE, iatrogenic pneumothoraces, and accidental punctures or lacerations. Their accuracy can be improved by strategies focused on adjustment of coding guidelines and efforts aimed at educating coders and ameliorating their clinical knowledge. In addition, introduction of “present on admission” flags in administrative data, an intervention already adopted in some states and in many hospitals as of 2007, will clearly improve the PPV of these PSIs by differentiating between new onset diagnoses and those that patients have before admission to the hospital. The ICD-9-CM codes used in the PSI algorithms were initially designed for billing purposes; therefore, using them for clinical and quality of care purposes may require modifications directly to the coding schemes. For example, coding all the “rule out” diagnoses referred to in a physician’s note when a postoperative patient experiences shortness of breath (eg, myocardial infarction, pulmonary embolus, pulmonary edema, COPD exacerbation, etc) might make sense from a financial perspective; however, it serves to decrease the specificities and alter the predictive values of the PSIs, which rely heavily on the clinical accuracy of the ICD-9-CM codes. Despite the relative lack of financial incentives in coding in the VA compared with other settings, the fact that our PPVs were similar to or lower than those in the private sector suggests that the shortcomings of the PSIs are inherent to the coding algorithms and practices, and not specific to any one setting. Shifting coding practices to rely on standard accepted clinical criteria rather than physician notes and reports alone will definitely improve the predictive value of PSIs. More importantly, tar-
Vol. 212, No. 6, June 2011
Kaafarani et al
Table 3. Analysis of True Cases of Accidental Punctures or Lacerations Variable
n Demographics Age, y, mean (SD) Gender, male, n (%) Race/ethnicity, n (%) White, non-Hispanic African American, non-Hispanic Hispanic Other/missing Body mass index, kg/m2, mean (SD) Comorbidities No. of comorbidities, mean (SD) No. of comorbidities, median (IQR) Specific comorbidities, n (%) Hypertension Diabetes mellitus Chronic pulmonary disease Congestive heart failure Liver disease Alcohol abuse Metastatic malignancy Weight loss Peripheral vascular disease Description of puncture or laceration Location of puncture or laceration Head, n (%) Neck, n (%) Chest, n (%) Abdomen, n (%) Upper extremity, n (%) Lower extremity, n (%) Type of puncture or laceration Vascular Gastrointestinal tract Abdominal organs Genitourinary tract Spine dura Serosal bowel tear Pleural injury Heart/lungs Miscellaneous Setting where puncture or laceration occurred Operating room Cardiac catheterization suite Emergency room Radiology suite Patient bedside Other/undetermined
Data
95 67 (10) 91 (96) 60 (63) 11 (12) 6 (6) 18 (19) 27.8 (5.5) 1.5 (1.2) 1 (2 [0–2]) 54 (57) 21 (22) 11 (12) 7 (7) 2 (2) 2 (2) 3 (3) 5 (5) 7 (7)
2 (2) 9 (9) 19 (20) 60 (63) 1 (1) 4 (4)
Validity of Selected Patient Safety Indicators
931
Table 3. Continued Variable
Level of training of person performing procedure, n (%) Attending physician Physician-in-training Physician assistant Unknown Risk factors present at the time of procedure Lysis of adhesions Presence of “scar” tissue Abnormal anatomy No. of accidental punctures or lacerations, n (%) 1 2 ⱖ3 Unknown Outcomes Length of stay, d Mean (SD) Median (range) All-cause mortality during admission, n (%)
Data
35 (37) 42 (44) 1 (1) 17 (18) 19 (20) 5 (5) 5 (5) 74 (78) 14 (17) 6 (6) 1 (1)
15 (22) 8 (1-168) 5 (5)
Numbers might not add to totals due to rounding. IQR, interquartile range.
geted coder education is essential to avoid many of the shortcomings of administrative data in general and the PSIs in specific. Misunderstanding of the nuances of clinical and surgical care was evident throughout this project, where physicians’ phrases were occasionally taken out of context, such as with the interpretation of PE as “pulmonary embolus” instead of “physical examination.” Even if the sensitivity and specificity of PSIs were at a theoretical 100%, one also needs to address the conceptual
24 (23) 18 (17) 16 (15) 14 (13) 11 (11) 6 (6) 5 (5) 4 (4) 6 (6) 75 (79) 4 (4) 1 (1) 2 (2) 1 (1) 12 (13) (continued)
Figure 4. Analysis of false positive cases of accidental punctures or lacerations (APLs). The percentages reported in the figure refer to the percentage of the total number of cases; those reported in the text of the manuscript refer to the percentage of the false positive cases only. POA, present on admission; r/o, rule out.
932
Kaafarani et al
Validity of Selected Patient Safety Indicators
framework that correlates the occurrence of an APL, iPTX, or pPE/DVT with suboptimal or poor quality of care. In other words, discussion of what actually makes a good quality indicator or measure goes beyond sensitivities and specificities; it implies a risk-adjusted causal relationship between the provision of certain health services and the occurrence of adverse patient outcomes. This discussion is certainly beyond the scope of this article, which is primarily concerned with examining the criterion validity of the PSIs. The interplay of patient characteristics, hospital/ physician/clinical team performance, and the specific circumstances surrounding an episode of care in defining outcomes is far from simple (even with risk adjustment), complicating identification of specific adverse events for the purpose of quality benchmarking and public reporting. Clinical implications
In addition to the policy implications of the results presented here, analyses of the true safety-related events (pPE/ DVT, iPTX, and APL) provide us with significant insights into the nature of the adverse events and the circumstances surrounding them. Postoperative pulmonary embolus and deep vein thrombosis
Orthopaedic and abdominal procedures accounted for more than half the cases in which a postoperative VTE occurred, stressing the importance of DVT mechanical and pharmacologic prophylaxis in noncardiac surgery in general, and particularly, in orthopaedic and abdominal procedures. It is also notable that 17% of the DVTs diagnosed did not involve lower extremity veins, a classical target for quality improvement efforts. The importance of such a finding stems from the fact that the mechanisms and preventive methods involved in upper versus lower extremity DVTs are essentially distinct. Although sequential compression devices, prophylactic anticoagulation, and early ambulation help prevent lower extremity vein thrombosis, timely removal of central venous access lines, such as central venous catheters or peripherally-inserted central catheters (PICC) lines, is more relevant in the prevention of upper extremity DVTs. We believe that risk-adjusted upper and lower extremity DVTs are both important as potential “quality measures;” therefore, modification of this specific PSI and associated ICD-9-CM codes to separately detect lower and upper extremity DVTs would be useful from a quality improvement perspective. The all-cause inpatient mortality of patients who sustained a VTE was elevated (19%), although it is extremely hard to tease out the mortality risk attributable to the VTE alone because many of these patients had multiple serious medical problems and complications during the same admission.
J Am Coll Surg
Iatrogenic pneumothorax
Interestingly, only 38% of the iatrogenic pneumothoraces detected could be attributed to central line insertion, and procedures such as cardiac pacemaker or defibrillator placement, transthoracic needle aspiration, and percutaneous liver biopsy accounted for a significant number of additional pneumothoraces. In addition, the mean body mass index of patients who sustained an iPTX was less than 25 kg/m2, and a large proportion of patients had airway disease, suggesting that a thin patient with COPD or emphysema might be at higher risk for an iPTX than an obese patient, who might present an otherwise technically challenging body habitus for insertion of a central line or a transthoracic needle drainage. Slightly less than half the procedures causing the pneumothoraces were performed by physicians-intraining, which raises the question of the adequacy of supervision of bedside or interventional procedures, especially central line placements. Accidental puncture or laceration
As expected, APLs were more common in the chest and abdomen and most commonly involved vascular and intraabdominal organs. In addition, it seems that this indicator detects a high proportion of injuries occurring during spinal operations (eg, durotomy). Similar to iPTX, a significant number of the injuries occurred when a procedure was being performed by a trainee rather than an attending physician. Such a finding is hard to interpret without conducting further research, but raises issues related to the importance of trainee supervision and the learning curve of complex surgical procedures. More importantly, the nature of the accidental punctures and lacerations detected by this indicator presents serious concerns with regard to the discriminatory ability of this indicator. At least 27% of these “true positive” cases revealed injuries that can be considered to have no real clinical (or surgical) relevance. Correlating an injury, such as a serosal bowel tear during an abdominal procedure characterized by extensive intra-abdominal adhesions, to the quality of care that the surgeon, the surgical team, or the hospital provides seems unfair at the least and even ridiculous at times. Adopting such a PSI before improving its ability to distinguish between clinically relevant and clinically irrelevant injuries will provide surgeons with incentives to avoid noting the occurrence of such minor injuries in their operative notes in fear of equating these with serious accidental injuries as indicated by the PSI. Such unintended consequences will clearly be counterproductive to efforts aimed at improving the quality of surgical care.
Vol. 212, No. 6, June 2011
Kaafarani et al
Limitations
This study is based on retrospective review of medical records of a limited sample of male veteran patients. One might argue that there is less financial incentive in the VA to strive for accuracy in coding in the VA system compared with the private sector, but previous studies show very good accuracy and reliability of diagnostic and procedural coding in the VA.24,25 In addition, we had limited objective ability to assess the preventability of the detected adverse events or their association with the quality of care provided. Nonetheless, if the study’s results are placed in the context of similar validation efforts taking place in several medical institutions across the nation, a reliable idea of the performance of these indicators can be achieved. The utility of these PSIs, particularly pPE/DVT, as performance measures, needs to be viewed with caution and should be reassessed. Future directions
As they currently stand, the use of some PSIs (particularly pPE/DVT) as quality measures for hospital safety profiling, public reporting, and pay-for performance is premature. Our research team is currently studying the sensitivity of pPE/DVT, iPTX, and APL and the processes of care that might have contributed to the occurrence of these adverse events; this task is necessary for a full evaluation of PSIs, but requires complex epidemiologic statistical models and extensive medical record review to detect safety-related events that are potentially missed by these indicators.
5.
6. 7. 8. 9. 10.
11. 12. 13.
Author Contributions Study conception and design: Kaafarani, Borzecki, Itani, Rosen Acquisition of data: Kaafarani, Borzecki, Loveland, Mull, Hickson, MacDonald, Shin, Rosen Analysis and interpretation of data: Kaafarani, Borzecki, Itani, Loveland, Mull, Shin, Rosen Drafting of manuscript: Kaafarani, Borzecki, Itani, Loveland, Mull, Hickson, MacDonald, Shin, Rosen Critical revision: Kaafarani, Borzecki, Itani, Rosen
14.
15.
16. 17.
REFERENCES 1. Institute of Medicine. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy; 2000. 2. Institute of Medicine. Crossing the Quality Chasm. Washington, DC: National Academy; 2001. 3. Zhan C, Miller MR. Excess length of stay, charges, and mortality attributable to medical injuries during hospitalization. JAMA 2003;290:1868–1874. 4. Miller DC, Filson CP, Wallner LP, et al. Comparing performance of Morbidity and Mortality Conference and National
18.
19.
Validity of Selected Patient Safety Indicators
933
Surgical Quality Improvement Program for detection of complications after urologic surgery. Urology 2006;68:931– 937. Hutter MM, Rowell KS, Devaney LA, et al. Identification of surgical complications and deaths: an assessment of the traditional surgical morbidity and mortality conference compared with the American College of Surgeons-National Surgical Quality Improvement Program. J Am Coll Surg 2006;203:618– 624. Romano PS, Geppert JJ, Davies S, et al. A national profile of patient safety in U.S. hospitals. Health Aff (Millwood) 2003;22: 154–166. AHRQ Quality Indicators: Guide to Patient Safety Indicators. Version 3.0a. Rockville, MD: Agency for Healthcare Research and Quality (AHRQ); March 2003; version 3.1. Healthgrades patient safety methodology. Available at http:// www.healthgrades.com/media/DMS/pdf/PatientSafety Methodology.pdf. Accessed April 15, 2010. National Quality Forum. NQF endorsed standards. Available at: http://www.qualityforum.org/Measures_List.aspx. Accessed April 15th, 2010. Centers for Medicare and Medicaid Services. Details for: Quality measures for reporting in fiscal year 2009 for 2010 update. Available at: http://www.cms.hhs.gov/apps/media/press/factsheet. asp?Counter⫽3044&intNumPerPage⫽10&checkDate⫽& checkKey⫽&srchType⫽1&numDays⫽3500&srchOpt⫽ 0&srchData⫽&keywordType⫽All&chkNewsType⫽6& intPage⫽&showAll⫽&pYear⫽&year⫽&desc⫽false&cbo Order⫽date. Accessed April 15, 2010. QualityNet. Available at: http://www.qualitynet.org. Accessed April 15, 2010. Bahl V, Thompson MA, Kau TY, et al. Do the AHRQ patient safety indicators flag conditions that are present at the time of hospital admission? Med Care 2008;46:516–522. Houchens RL, Elixhauser A, Romano PS. How often are potential patient safety events present on admission? Jt Comm J Qual Patient Saf 2008;34:154–163. White RH, Sadeghi B, Tancredi DJ, et al. How valid is the ICD-9-CM based AHRQ patient safety indicator for postoperative venous thromboembolism? Med Care 2009;47:1237– 1243. Henderson KE, Recktenwald A, Reichley RM, et al. Clinical validation of the AHRQ postoperative venous thromboembolism patient safety indicator. Jt Comm J Qual Patient Saf 2009; 35:370–376. Utter GH, Zrelak PA, Baron R, et al. Positive predictive value of the AHRQ accidental puncture or laceration patient safety indicator. Ann Surg 2009;250:1041–1045. Weller WE, Gallagher BK, Cen L, Hannan EL. Readmissions for venous thromboembolism: expanding the definition of patient safety indicators. Jt Comm J Qual Saf 2004;30:497– 504. Gallagher B, Cen L, Hannan EL. Validation of AHRQ’s patient safety indicator for accidental puncture or laceration. Available at: http://www.ncbi.nlm.nih.gov/bookshelf/br.fcgi?book⫽ aps2&part⫽A1586. Accessed April 15, 2010. Zhan C, Battles J, Chiang YP, Hunt D. The validity of ICD9-CM codes in identifying postoperative deep vein thrombosis and pulmonary embolism. Jt Comm J Qual Patient Saf 2007; 33:326–331.
934
Kaafarani et al
Validity of Selected Patient Safety Indicators
20. Kaafarani HM, Rosen AK. Using administrative data to identify surgical adverse events: an introduction to the Patient Safety Indicators. Am J Surg 2009;198:S63–68. 21. Quality AfHRa. Patient Safety Indicators: technical specifications, March 2003, version 3.1. 22. Sadeghi B, Baron R, Zrelak P, et al. Cases of iatrogenic pneumothorax can be identified from ICD-9-CM coded data. Am J Med Qual 2010;25:218–224.
J Am Coll Surg
23. Allison JJ, Wall TC, Spettell CM, et al. The art and science of chart review. Jt Comm J Qual Improv 2000;26:115–136. 24. Kashner TM. Agreement between administrative files and written medical records: a case of the Department of Veterans Affairs. Med Care 1998;36:1324–1336. 25. Petersen LA, Wright S, Normand SL, Daley J. Positive predictive value of the diagnosis of acute myocardial infarction in an administrative database. J Gen Intern Med 1999;14:555–558.
health care priority.1,2 The number of studies aimed at measuring adverse events, safety-related events, errors, or patient harm has increased exponentially in the last few years. Medical record review promises richer clinical detail in identifying adverse events, but is resource-intensive and expensive.3 Traditional mortality and morbidity conferences show less than 25% sensitivity in detecting adverse events when compared with the large chart-based quality measurement initiatives such as the National Surgical Quality Improvement Program (NSQIP).4,5 Recently, in an effort to improve and increase detection of potentially preventable safety events in acute-care hospitals, the Agency for Healthcare Research and Quality (AHRQ) designed a set of evidence-based ICD-9-CMbased algorithms called Patient Safety Indicators (PSIs). The PSIs represent a significant advance in the field of patient safety because they use readily available hospital discharge data that can be risk adjusted to screen for safetyrelated events in the inpatient setting.6 The PSIs were ini-
Since publication of the Institute of Medicine’s 2 landmark reports, “To Err is Human” and “Crossing the Quality Chasm” in 2000, patient safety has become a national Disclosure Information: Nothing to disclose. Abstract presented American College of Surgeons 95th Annual Clinical Congress, Surgical Forum, Chicago, IL, October 2009. Received April 19, 2010; Revised July 8, 2010; Accepted July 8, 2010. From the Department of Surgery, Tufts Medical Center, Tufts University School of Medicine (Kaafarani), VA Boston Healthcare System and the Center for Health Quality, Outcomes and Economic Research (Kaafarani), Boston University School of Public Health (Borzecki, Mull, Rosen), Boston University School of Medicine (Borzecki, Itani), Harvard Medical School (Itani), and the Center for Organization, Leadership and Management Research, VA Boston Healthcare System (Loveland, Mull, Shin, Rosen), Boston, MA; the Center for Health Quality, Outcomes and Economic Research, Bedford VAMC, Bedford, MA (Borzecki, Hickson, MacDonald); and the Department of Surgery, VA Boston Healthcare System, West Roxbury, MA (Itani, Hickson). Correspondence address: Amy K Rosen, PhD, Center for Organization, Leadership and Management Research (COLMR), VA Boston Healthcare System, 150 S Huntington Ave, Boston MA 02130. email: [email protected] or [email protected]
© 2011 by the American College of Surgeons Published by Elsevier Inc.
924
ISSN 1072-7515/11/$36.00 doi:10.1016/j.jamcollsurg.2010.07.007
Vol. 212, No. 6, June 2011
Kaafarani et al
Abbreviations and Acronyms
AHRQ APL CMS iPTX IRR pPE/DVT
⫽ ⫽ ⫽ ⫽ ⫽ ⫽
PPV PSI VA VTE
⫽ ⫽ ⫽ ⫽
Agency for Healthcare Research and Quality accidental puncture and laceration Center for Medicare and Medicaid Services iatrogenic pneumothorax inter-rater reliability postoperative pulmonary embolus and deep vein thrombosis positive predictive value Patient Safety Indicator Veterans Health Administration venous thromboembolism
tially intended as screening tools to identify safety-related events or as case-finding tools for internal quality improvement purposes. However, in the last several years, they have been used by multiple organizations for hospital profiling and pay-for-performance purposes.7,8 Eight of the PSIs have been endorsed by the National Quality Forum (NQF) as hospital performance measures, and 4 (plus 1 composite measure) have been adopted by the Centers for Medicare and Medicaid Services (CMS) for hospital comparisons of quality and safety and financial reimbursement.9,10 With the increased use of PSIs for quality assessment, public reporting, hospital profiling, and reimbursement,8,11 many clinicians, policymakers, and researchers have raised concerns over the validity of PSIs, given that the PSI algorithms are based on administrative data that are well known for their variability and inconsistency in the coding of diagnoses and procedures. Some of these coding issues include ambiguity in ICD-9-CM coding guidelines, variation in coding practices across different hospitals, and the codes’ inability to differentiate between events that happened de novo versus those that were present on admission.12,13 Although the current literature suggests that PSIs examined to date have moderate to high sensitivities and specificities, their measured performance has been found to depend on the nature of the specific PSI, the nature of the adverse events it targets, and the method of validation used to test its performance. In these multiple validation attempts, the calculated positive predictive values (PPVs) of PSIs against chart abstraction as the “gold standard,” ranged between 44% and 91%.14-20 Although the criterion validity of several PSIs against the gold standard has been examined in the private sector, little is known about whether these results are reliable in other health care settings, such as the Veterans Health Administration (VA). In this study, we examined the positive PPV of 3 surgical PSIs against medical record review: postoperative pulmonary embolus and deep vein thrombosis (pPE/ DVT), iatrogenic pneumothorax (iPTX), and accidental
Validity of Selected Patient Safety Indicators
925
puncture and laceration (APL). These 3 PSIs have all been endorsed by the National Quality Forum as quality measures; APL and iPTX have been adopted by CMS.
METHODS We applied the AHRQ PSI software (v.3.1a) to VA fiscal year 2003 to 2007 (October 1, 2002 to September 30, 2007) administrative data from a sample of 28 VA hospitals to identify (flag) patients suspected of having a pPE/DVT, iPTX, or APL. To determine rates of true and false positives for each PSI, trained nurses conducted a retrospective chart review of 336 flagged charts (112 charts per PSI) using standardized chart abstraction tools and guidelines developed by the AHRQ and modified for the VA’s electronic medical record. Inter-rater reliability testing of the 2 nurse abstractors was also performed. The required Institutional Review Board (IRB) approvals from the Bedford VA Medical Center and the VA Boston Healthcare System were obtained to conduct this study. PSI overview and definitions
PSIs are calculated as rates: the numerator consists of all discharges with diagnoses, procedures, and/or other attributes indicating the occurrence of the particular adverse event; the denominator includes all discharges at risk for the adverse event, with certain exclusions. All denominators include patients aged 18 and older and exclude patients with Major Diagnostic Category (MDC) 14 (pregnancy, childbirth, and puerperium).21 Postoperative PE/DVT detects the occurrence of postoperative venous thromboembolism (VTE) by identifying discharges that have any of the ICD-9-CM codes for PE or DVT in a secondary diagnosis field. The denominator includes all surgical discharges as defined by specific Diagnosis-Related Groups (DRGs) and an ICD-9-CM code for an operating room procedure, narrowing the denominator to those cases most likely to be preventable. It excludes all patients with pre-existing (principal diagnosis or secondary diagnosis present on admission, if known) PE or DVT and patients who underwent a procedure to interrupt the vena cava either as the only operating room procedure during admission or before or on the same day of the index operating room procedure. The occurrence of iPTX is targeted by identifying discharges with the ICD-9-CM code 512.1 in any secondary diagnosis field. The denominator includes all surgical and medical discharges of patients with specific DRGs. It excludes patients with the code 512.1 in a primary diagnosis field; a code for diaphragmatic surgery repair; a cardiac surgery DRG; a thoracic surgery or lung or pleural biopsy code; and a diagnosis code for chest trauma or pleural ef-
926
Kaafarani et al
Validity of Selected Patient Safety Indicators
J Am Coll Surg
Figure 1. Hospital sampling strategy. PSI, Patient Safety Indicator.
fusion. Iatrogenic pneumothoraces caused by procedures performed within 72 hours before admission are included. Similar to pPE/DVT, the denominator was purposely restrictive in order to capture the most “preventable” cases. APL captures the occurrence of accidental cut, puncture, perforation, or hemorrhage during medical care (ICD9-CM codes E870.0 through E870.9) or accidental puncture or laceration during a procedure (ICD-9-CM code 998.2) in a secondary diagnosis field. The denominator includes all surgical and medical discharges excluding patients whose APL was present on admission or in a primary diagnosis field.
the expected and observed numerators in any of the PSIs were excluded, yielding a final sample of 79 hospitals. Within each tier of hospitals, facilities were ranked using the AHRQ PSI composite rate.21 The top 3 and bottom 3 hospitals were included in the hospital sample. We then randomly selected from the remaining hospitals within each stratum to obtain a sample of 28 hospitals. To assure balanced geographic representation, 3 hospitals were replaced by the next hospital in rank, for a final sample of 28 hospitals representing diverse geographic regions of the US. The hospital selection process is shown in Figure 1. Case selection
Hospital selection
Our initial hospital sample included 158 acute-care VA hospitals. To obtain a manageable number of hospitals for chart review, and to minimize variation in coding across hospitals, we selected a sample of hospitals from the 158 that represented a broad spectrum of PSI rates. We grouped the 158 hospitals into 3 tiers based on their observed and expected rates of PSIs, exclusive of PSI 5 (foreign body left during procedure) and PSI 8 (postoperative hip fracture), both of which had low incidence rates across most hospitals. The expected number of PSI events of a specific facility was calculated as the national VA PSI rate multiplied by the PSI denominator of that specific facility. The first group of hospitals included facilities that had a numerator of at least 4 safety-related events in both the expected and observed numerators of each PSI. The second group had at least 2 safety-related events in the expected and observed numerators of each PSI. The third group had at least 1 safetyrelated event in the expected and observed numerators of each PSI. Hospitals with less than 1 safety-related event in
Four flagged medical records per PSI (pPE/DVT, iPTX, APL) were randomly selected from each of the 28 hospitals for a total of 336 medical records. Based on previously reported PPV estimates, 112 cases per PSI were selected to ensure reasonably narrow PPV confidence intervals (range 10% to 20%). When a certain facility had less than 4 flagged records for a specific PSI, flagged records were reviewed from the next facility within the stratum, for a maximum of 8 records total per hospital. Medical record abstraction
Two trained nurses used standardized abstraction instruments to review medical records for occurrence of a safetyrelated event (pPE/DVT, iPTX, or APL); demographics, comorbidities, and risk factors of the patient population; clinical circumstances surrounding the safety-related event; and patient outcomes after the event. Nurse training included several sessions discussing the rationale behind each PSI, the likely sources of information needed from the electronic medical record, and a systematic chronology for
Vol. 212, No. 6, June 2011
Kaafarani et al
chart abstraction. When the nurse abstractors were uncertain about any item, the medical record was referred to the research team physicians (HK, AB, KI) for resolution.
Validity of Selected Patient Safety Indicators
927
RESULTS Of 2,343,088 admissions, 6,080 were flagged for pPE/ DVT (0.26%), 1,402 for iPTX (0.06%), and 7,203 for APL (0.31%).
Inter-rater reliability
After completion of nurse training, we initiated inter-rater reliability (IRR) testing, as recommended in the literature, to obtain a standardized and reliable method of abstraction.22 At least 10% of the medical records were reviewed by both nurses, and IRR was measured as the percentage of agreement on select key questions of each abstraction tool, such as ascertainment of the safety-related event. Records were abstracted in groups of 5 until ⬎90% agreement was obtained; this typically occurred after the first or second round of IRR, and then nurses were instructed to proceed by abstracting separate charts. After each round of IRR (including the round achieving ⬎90%), a discussion of disagreements took place in the presence of the clinical physicians (HK, AB, KI) with resulting instrument revisions and/or guideline clarifications as appropriate. An additional round of IRR was performed on 5 records toward the end of the abstraction process to check for potential abstractor drift. Positive predictive value
For each of the 3 PSIs, we calculated PPV as the rate of true positives divided by the number of medical records reviewed and derived 95% confidence intervals for that estimate.
Postoperative pulmonary embolus or deep vein thrombosis Positive predictive value
Of 112 cases, 48 were true events of postoperative PE or DVT, yielding a PPV of 43% (95% CI 34% to 53%). IRR between chart abstractors was measured at 94%. True positive analysis
As shown in Table 1, the patient population was entirely male, with a mean age of 70 years. Seventy-one percent were white, non-Hispanic patients. The mean number of comorbidities per patient was 1.6: 17% of the patients were diabetic, 17% had chronic pulmonary disease, 13% had a malignancy, and 8% had congestive heart failure. Twentytwo of the 48 true pPE/DVT patients had only a DVT diagnosis (46%), 19 patients had only a PE diagnosis (40%), and 7 patients had both a DVT and a PE diagnosed (15%). Of the total of 29 DVTs, 23 involved the lower extremity (79%), and only 4 involved the upper extremities (14%). Orthopaedic and abdominal procedures accounted for more than half of the index procedures preceding the VTE events (38% and 21%, respectively); 67% of the index procedures were elective in nature. The median length of stay for these admissions was 22 days. All-cause inhospital mortality was 19%.
True positive analysis
For patients with a confirmed pPE/DVT, iPTX, or APL, we performed descriptive analyses of multiple continuous and categorical variables including demographics (age, gender, and race or ethnicity), comorbidities, relevant risk factors, and the nature of the surgical procedure and outcomes. For example, for pPE/DVT, the location of the DVT and the index procedure preceding the PE or DVT were identified. For iPTX, we examined the cause of the pneumothorax (eg, subclavian or internal jugular central line insertion, ventilator-associated barotrauma) and the training level of the staff involved in the procedure (attending vs trainee). For APL, we examined the nature of the surgical procedure, the type of puncture or laceration (eg, bowel injury, durotomy, and splenic injury), and the training level of the staff primarily involved in the procedure. All statistical analyses were performed using SAS version 9.1. False positive analysis
All false positive cases underwent further detailed review to better understand why they were incorrectly flagged by the PSI algorithms.
False positive analysis
Of a total of 64 false positive cases, 16 (25%) patients had a PE and/or DVT diagnosis present on admission (diagnosed 6 months or less before admission), 10 patients (16%) had a remote history of PE/DVT (diagnosed more than 6 months before admission), and 14 patients (22%) were diagnosed after admission but before the index procedure. Coding-related inaccuracies accounted for the remaining 24 false positives (38%) (Fig. 2). These included cases of arterial (not venous) thrombosis, superficial (not deep) vein thrombosis or thrombophlebitis, and cases in which a postoperative PE and DVT workup was negative or the etiology for patient postoperative mortality was uncertain (ie, “rule out” PE). Ten of the cases (9%) were considered false positives for diverse reasons (the miscellaneous category). For example, one of the false positive cases had a discharge summary stating: “patient with history and PE compatible with acute appendicitis.” It seems likely that, in this instance, the coder mistook the abbreviation “PE” to stand for pulmonary embolus rather than physical examination.
928
Kaafarani et al
Validity of Selected Patient Safety Indicators
Table 1. Analysis of True Positives for Postoperative Pulmonary Embolus or Deep Vein Thrombosis Variable
n Demographics Age, y, mean (SD) Gender, male, n (%) Race/ethnicity, n (%) White, non-Hispanic African American, non-Hispanic Hispanic Other/missing Body mass index, kg/m2, mean (SD) Comorbidities No. of comorbidities, mean (SD) No. of comorbidities, median (IQR) Specific comorbidities, n (%) Diabetes mellitus Chronic pulmonary disease Congestive heart failure Renal failure Solid malignancy without metastasis Metastatic malignancy Lymphoma Obesity Paralysis Other neurologic disorders Additional risk factors, n (%) Hypercoagulable state Baseline inability to ambulate History of spinal cord injury History of recent trauma Central venous catheter insertion Femoral Subclavian Internal jugular Peripherally inserted central line Ventilator dependence Chemotherapy Use of potentially procoagulant medication Transfusion of blood or blood products Index procedures, n (%) Elective procedure Procedure type Orthopaedic Abdominal Urologic (including nephrectomy) Thoracic/pulmonary Cardiac Neurosurgical Vascular
Data
48 70 (11) 48 (100) 34 (71) 5 (10) 3 (6) 6 (13) 27.1 (4.7) 1.6 (1.3) 1 (1.5 [1–2.5]) 8 (17) 8 (17) 4 (8) 3 (6) 3 (6) 2 (4) 1 (2) 1 (2) 1 (2) 2 (4) 2 (4) 5 (10) 1 (2) 0 (0) 1 (2) 5 (10) 7 (15) 13 (27) 8 (17) 2 (4) 0 (0) 0 (0)
J Am Coll Surg
Table 1. Continued Variable
Type of venous thromboembolism, n (%) Pulmonary embolus only Deep vein thrombosis only Both pulmonary embolus and deep vein thrombosis Location of deep vein thrombosis, when applicable, n⫽ 29, n (%) Lower extremity Upper extremity Internal jugular Undocumented location Outcomes Length of stay, d Mean (SD) Median (range) All-cause mortality during admission, n (%)
Data
19 (40) 22 (46) 7 (15)
23 (79) 4 (14) 1 (3) 1 (3)
35 (32) 22 (1–127) 9 (19)
Numbers might not add to totals due to rounding. IQR, interquartile range.
Iatrogenic pneumothorax Positive predictive value
Out of 112 cases reviewed, 82 were true iatrogenic pneumothoraces, yielding a PPV of 73% (95% CI, 64% to 81%]. IRR between the chart abstractors was measured at 94%. True positive analysis
As shown in Table 2, the population was 94% male and 62% white non-Hispanic; mean age was 68 years, and the mean body mass index was 24.5 kg/m2. Hypertension, airway lung disease (eg, asthma, COPD), and diabetes mel-
32 (67) 18 (38) 10 (21) 6 (13) 6 (13) 3 (6) 3 (6) 2 (4) (continued)
Figure 2. Positive predictive value and analysis of false positives of postoperative pulmonary embolus or deep vein thrombosis (pPE/ DVT). Numbers might not add to totals due to rounding. The percentages reported in the figure refer to the percentage of the total number of cases; those reported in the text of the manuscript refer to the percentage of the false positive cases only.
Vol. 212, No. 6, June 2011
Kaafarani et al
Table 2. Analysis of True Positives for Iatrogenic Pneumothorax Variable
n Demographics Age, y, mean (SD) Gender, male, n (%) Race/ethnicity, n (%) White, non-Hispanic African American, non-Hispanic Hispanic Other/missing Body mass index, kg/m2, mean (SD) Comorbidities No. of comorbidities, mean (SD) No. of comorbidities, median (IQR) Specific comorbidities, n (%) Hypertension Diabetes mellitus Chronic pulmonary disease Congestive heart failure Weight loss Alcohol abuse Liver disease Metastatic malignancy Peripheral vascular disease Pulmonary-related risk factors, n (%) Airway disease (eg, COPD, asthma) Respiratory tract malignancy Infectious lung disease (eg, pneumonia) Pleural effusion Interstitial lung disease (eg, lung fibrosis) Other respiratory disease Description of the iatrogenic pneumothorax Causes of the pneumothorax, n (%)* Central venous catheter insertion Transthoracic needle aspiration (or biopsy) Cardiac pacemaker placement Implantable defibrillator insertion Mechanical ventilation (barotrauma) Liver biopsy/liver lesion radiofrequency ablation Cardiopulmonary resuscitation Miscellaneous procedures near chest/neck Level of training of person performing procedure, n (%), n ⫽ 85 Attending Physician-in-training Unknown Outcomes Length of stay, d Mean (SD) Median (range) All-cause mortality during admission, n (%) Numbers might not add to totals due to rounding. *Three patients had 2 potential causative procedures each. IQR, interquartile range.
Data
82 68 (14) 77 (94) 51 (62) 12 (15) 4 (5) 15 (18) 24.5 (6.5) 1.7 (1.3) 2 (1 [1–2]) 31 (38) 19 (23) 19 (23) 12 (15) 6 (7) 4 (5) 2 (2) 2 (2) 2 (2) 24 (29) 9 (11) 8 (10) 7 (8) 3 (4) 5 (6)
31 (38) 15 (18) 11 (13) 4 (5) 3 (4) 3 (4) 1 (1) 14 (17)
28 (33) 38 (45) 19 (22)
18 (21) 12 (1–140) 15 (18)
Validity of Selected Patient Safety Indicators
929
litus were the most prevalent risk factors in the patient population (38%, 29%, and 23%, respectively). Of 82 cases of true iatrogenic pneumothoraces, 31 were caused by central venous catheter insertions (38%), 15 by transthoracic needle aspirations or biopsies (18%), 15 by cardiac pacemaker or defibrillator placement (18%), 3 by liver biopsies or liver lesion radiofrequency ablation (4%), 3 by mechanical ventilation (4%), and 1 by cardiopulmonary resuscitation (1%). Several miscellaneous procedures (eg, brachial, axillary or intercostal nerve blocks, pericardiocentesis) accounted for 14 pneumothoraces (17%). Forty-five percent of the pneumothoraces occurred during procedures performed by physicians in training; 33% occurred during procedures performed by attending-level physicians alone. The median length of hospital stay for these patients was 12 days, and the all-cause in-hospital mortality was 18%. False positive analysis
There were 30 false positive cases, including 10 patients with an old history of pneumothorax or pneumothorax present on admission (more than 72 hours before admission), and 20 cases of inaccurate coding (Fig 3). The latter included cases in which the pneumothorax was spontaneous but not iatrogenic; cases in which lung consolidation or collapse due to pneumonia or empyema were mistaken for iatrogenic pneumothoraces; and cases in which the procedure was known a priori to breach the pleural cavity, such as a thoracic procedure or a lung biopsy. Accidental puncture or laceration Positive predictive value
Ninety-five of 112 cases represented “true” cases of accidental punctures or lacerations, yielding a PPV of 85% (95% CI, 77% to 91%). IRR between the chart abstractors was measured at 97%. True positive analysis
As illustrated in Table 3, the population was 96% male and 63% white non-Hispanic; mean age was 67 years, and mean body mass index was 27.8 kg/m2. The mean number of comorbidities was 1.5, with the most common comorbidities being hypertension (57%), diabetes mellitus (22%), and chronic pulmonary disease (12%). Seventynine of the 95 true iatrogenic events involved the chest or the abdomen (83%); 75 of 95 occurred in the operating room (79%). Injury to vascular structures (inadvertent injury to large arteries or veins), bowel (eg, iatrogenic enterotomy), abdominal organs (eg, splenic injury in a colectomy), genitourinary organs (eg, ureteral injury), and spinal dura (eg, durotomy) accounted for 81% of the punctures or lacerations (23%, 17%, 15%, 13%, and 11%,
930
Kaafarani et al
Validity of Selected Patient Safety Indicators
Figure 3. Positive predictive value and analysis of false positives of iatrogenic pneumothorax (iPTX). Numbers might not add to totals due to rounding. The percentages reported in the figure refer to the percentage of the total number of cases; those reported in the text of the manuscript refer to the percentage of the false positive cases only. POA, present on admission.
respectively). Simple serosal bowel tears accounted for 6% of the injuries. When management of these injuries was examined, 26% of them required no surgical management at all (eg, a small serosal bowel tear that was not repaired or a small spinal durotomy that was managed using gelfoam only). Forty-four percent of these injuries occurred during procedures performed by physicians in training; 37% occurred during procedures performed by attending-level physicians alone. In 25% of cases, surgeons reported presence of adhesions or scar tissue; an additional 5% were associated with abnormal anatomy. The median length of stay for patients who sustained these injuries was 8 days. The all-cause in-hospital mortality of these patients was 5%. False positive analysis
Out of a total of 17 false positive cases, 6 were patients with punctures or lacerations that were present on admission (35%) and 11 were cases of inaccurate coding diagnosis (65%) (Fig. 4). Half of the cases in the latter category were due to the puncture or laceration being nonaccidental (eg, ampulla of Vater “slit” during an endoscopic retrograde cholangiopancreatography to release impacted stones; spontaneous nonprocedure sigmoid colon perforation). The other half involved miscellaneous conditions (eg, suspicion of an intubation-related pharyngeal injury that was later ruled out), including cases in which the reason for coding as an accidental puncture or laceration remained unclear to the chart abstractors.
DISCUSSION This is the first study to examine the validity of pPE/DVT, iPTX, and APL in the VA. We found that the PPVs of these
J Am Coll Surg
3 PSIs varied considerably. Postoperative PE/DVT had a low PPV (43%), iPTX had a moderate PPV (74%), and APL had a moderate-to-high PPV (85%). However, 26% of the accidental punctures or lacerations detected by the APL algorithm were minor injuries that did not require any surgical repair or intervention (eg, serosal bowel tears or dural tears that heal with simple observation or with the use of sealants). Our PPV findings are consistent with those from private sector studies.14-20,23 Two recent studies estimated the PPV of pPE/DVT to be between 44% and 55%.14 A separate study estimated the PPV of APL to be around 91%, but similarly noted that 24% of the injuries were “inconsequential,” “expected to heal without repair,” and for which “the risk may have been acceptable relative to the goals of the procedure.”16 Another recent study estimated the PPV of iPTX to be around 78%, similar to our findings, with 44% of the pneumothoraces resulting from central venous line insertions.23 Policy implications
The PSIs we studied show promise as screening tools that can be used to detect patient safety events related to postoperative VTE, iatrogenic pneumothoraces, and accidental punctures or lacerations. Their accuracy can be improved by strategies focused on adjustment of coding guidelines and efforts aimed at educating coders and ameliorating their clinical knowledge. In addition, introduction of “present on admission” flags in administrative data, an intervention already adopted in some states and in many hospitals as of 2007, will clearly improve the PPV of these PSIs by differentiating between new onset diagnoses and those that patients have before admission to the hospital. The ICD-9-CM codes used in the PSI algorithms were initially designed for billing purposes; therefore, using them for clinical and quality of care purposes may require modifications directly to the coding schemes. For example, coding all the “rule out” diagnoses referred to in a physician’s note when a postoperative patient experiences shortness of breath (eg, myocardial infarction, pulmonary embolus, pulmonary edema, COPD exacerbation, etc) might make sense from a financial perspective; however, it serves to decrease the specificities and alter the predictive values of the PSIs, which rely heavily on the clinical accuracy of the ICD-9-CM codes. Despite the relative lack of financial incentives in coding in the VA compared with other settings, the fact that our PPVs were similar to or lower than those in the private sector suggests that the shortcomings of the PSIs are inherent to the coding algorithms and practices, and not specific to any one setting. Shifting coding practices to rely on standard accepted clinical criteria rather than physician notes and reports alone will definitely improve the predictive value of PSIs. More importantly, tar-
Vol. 212, No. 6, June 2011
Kaafarani et al
Table 3. Analysis of True Cases of Accidental Punctures or Lacerations Variable
n Demographics Age, y, mean (SD) Gender, male, n (%) Race/ethnicity, n (%) White, non-Hispanic African American, non-Hispanic Hispanic Other/missing Body mass index, kg/m2, mean (SD) Comorbidities No. of comorbidities, mean (SD) No. of comorbidities, median (IQR) Specific comorbidities, n (%) Hypertension Diabetes mellitus Chronic pulmonary disease Congestive heart failure Liver disease Alcohol abuse Metastatic malignancy Weight loss Peripheral vascular disease Description of puncture or laceration Location of puncture or laceration Head, n (%) Neck, n (%) Chest, n (%) Abdomen, n (%) Upper extremity, n (%) Lower extremity, n (%) Type of puncture or laceration Vascular Gastrointestinal tract Abdominal organs Genitourinary tract Spine dura Serosal bowel tear Pleural injury Heart/lungs Miscellaneous Setting where puncture or laceration occurred Operating room Cardiac catheterization suite Emergency room Radiology suite Patient bedside Other/undetermined
Data
95 67 (10) 91 (96) 60 (63) 11 (12) 6 (6) 18 (19) 27.8 (5.5) 1.5 (1.2) 1 (2 [0–2]) 54 (57) 21 (22) 11 (12) 7 (7) 2 (2) 2 (2) 3 (3) 5 (5) 7 (7)
2 (2) 9 (9) 19 (20) 60 (63) 1 (1) 4 (4)
Validity of Selected Patient Safety Indicators
931
Table 3. Continued Variable
Level of training of person performing procedure, n (%) Attending physician Physician-in-training Physician assistant Unknown Risk factors present at the time of procedure Lysis of adhesions Presence of “scar” tissue Abnormal anatomy No. of accidental punctures or lacerations, n (%) 1 2 ⱖ3 Unknown Outcomes Length of stay, d Mean (SD) Median (range) All-cause mortality during admission, n (%)
Data
35 (37) 42 (44) 1 (1) 17 (18) 19 (20) 5 (5) 5 (5) 74 (78) 14 (17) 6 (6) 1 (1)
15 (22) 8 (1-168) 5 (5)
Numbers might not add to totals due to rounding. IQR, interquartile range.
geted coder education is essential to avoid many of the shortcomings of administrative data in general and the PSIs in specific. Misunderstanding of the nuances of clinical and surgical care was evident throughout this project, where physicians’ phrases were occasionally taken out of context, such as with the interpretation of PE as “pulmonary embolus” instead of “physical examination.” Even if the sensitivity and specificity of PSIs were at a theoretical 100%, one also needs to address the conceptual
24 (23) 18 (17) 16 (15) 14 (13) 11 (11) 6 (6) 5 (5) 4 (4) 6 (6) 75 (79) 4 (4) 1 (1) 2 (2) 1 (1) 12 (13) (continued)
Figure 4. Analysis of false positive cases of accidental punctures or lacerations (APLs). The percentages reported in the figure refer to the percentage of the total number of cases; those reported in the text of the manuscript refer to the percentage of the false positive cases only. POA, present on admission; r/o, rule out.
932
Kaafarani et al
Validity of Selected Patient Safety Indicators
framework that correlates the occurrence of an APL, iPTX, or pPE/DVT with suboptimal or poor quality of care. In other words, discussion of what actually makes a good quality indicator or measure goes beyond sensitivities and specificities; it implies a risk-adjusted causal relationship between the provision of certain health services and the occurrence of adverse patient outcomes. This discussion is certainly beyond the scope of this article, which is primarily concerned with examining the criterion validity of the PSIs. The interplay of patient characteristics, hospital/ physician/clinical team performance, and the specific circumstances surrounding an episode of care in defining outcomes is far from simple (even with risk adjustment), complicating identification of specific adverse events for the purpose of quality benchmarking and public reporting. Clinical implications
In addition to the policy implications of the results presented here, analyses of the true safety-related events (pPE/ DVT, iPTX, and APL) provide us with significant insights into the nature of the adverse events and the circumstances surrounding them. Postoperative pulmonary embolus and deep vein thrombosis
Orthopaedic and abdominal procedures accounted for more than half the cases in which a postoperative VTE occurred, stressing the importance of DVT mechanical and pharmacologic prophylaxis in noncardiac surgery in general, and particularly, in orthopaedic and abdominal procedures. It is also notable that 17% of the DVTs diagnosed did not involve lower extremity veins, a classical target for quality improvement efforts. The importance of such a finding stems from the fact that the mechanisms and preventive methods involved in upper versus lower extremity DVTs are essentially distinct. Although sequential compression devices, prophylactic anticoagulation, and early ambulation help prevent lower extremity vein thrombosis, timely removal of central venous access lines, such as central venous catheters or peripherally-inserted central catheters (PICC) lines, is more relevant in the prevention of upper extremity DVTs. We believe that risk-adjusted upper and lower extremity DVTs are both important as potential “quality measures;” therefore, modification of this specific PSI and associated ICD-9-CM codes to separately detect lower and upper extremity DVTs would be useful from a quality improvement perspective. The all-cause inpatient mortality of patients who sustained a VTE was elevated (19%), although it is extremely hard to tease out the mortality risk attributable to the VTE alone because many of these patients had multiple serious medical problems and complications during the same admission.
J Am Coll Surg
Iatrogenic pneumothorax
Interestingly, only 38% of the iatrogenic pneumothoraces detected could be attributed to central line insertion, and procedures such as cardiac pacemaker or defibrillator placement, transthoracic needle aspiration, and percutaneous liver biopsy accounted for a significant number of additional pneumothoraces. In addition, the mean body mass index of patients who sustained an iPTX was less than 25 kg/m2, and a large proportion of patients had airway disease, suggesting that a thin patient with COPD or emphysema might be at higher risk for an iPTX than an obese patient, who might present an otherwise technically challenging body habitus for insertion of a central line or a transthoracic needle drainage. Slightly less than half the procedures causing the pneumothoraces were performed by physicians-intraining, which raises the question of the adequacy of supervision of bedside or interventional procedures, especially central line placements. Accidental puncture or laceration
As expected, APLs were more common in the chest and abdomen and most commonly involved vascular and intraabdominal organs. In addition, it seems that this indicator detects a high proportion of injuries occurring during spinal operations (eg, durotomy). Similar to iPTX, a significant number of the injuries occurred when a procedure was being performed by a trainee rather than an attending physician. Such a finding is hard to interpret without conducting further research, but raises issues related to the importance of trainee supervision and the learning curve of complex surgical procedures. More importantly, the nature of the accidental punctures and lacerations detected by this indicator presents serious concerns with regard to the discriminatory ability of this indicator. At least 27% of these “true positive” cases revealed injuries that can be considered to have no real clinical (or surgical) relevance. Correlating an injury, such as a serosal bowel tear during an abdominal procedure characterized by extensive intra-abdominal adhesions, to the quality of care that the surgeon, the surgical team, or the hospital provides seems unfair at the least and even ridiculous at times. Adopting such a PSI before improving its ability to distinguish between clinically relevant and clinically irrelevant injuries will provide surgeons with incentives to avoid noting the occurrence of such minor injuries in their operative notes in fear of equating these with serious accidental injuries as indicated by the PSI. Such unintended consequences will clearly be counterproductive to efforts aimed at improving the quality of surgical care.
Vol. 212, No. 6, June 2011
Kaafarani et al
Limitations
This study is based on retrospective review of medical records of a limited sample of male veteran patients. One might argue that there is less financial incentive in the VA to strive for accuracy in coding in the VA system compared with the private sector, but previous studies show very good accuracy and reliability of diagnostic and procedural coding in the VA.24,25 In addition, we had limited objective ability to assess the preventability of the detected adverse events or their association with the quality of care provided. Nonetheless, if the study’s results are placed in the context of similar validation efforts taking place in several medical institutions across the nation, a reliable idea of the performance of these indicators can be achieved. The utility of these PSIs, particularly pPE/DVT, as performance measures, needs to be viewed with caution and should be reassessed. Future directions
As they currently stand, the use of some PSIs (particularly pPE/DVT) as quality measures for hospital safety profiling, public reporting, and pay-for performance is premature. Our research team is currently studying the sensitivity of pPE/DVT, iPTX, and APL and the processes of care that might have contributed to the occurrence of these adverse events; this task is necessary for a full evaluation of PSIs, but requires complex epidemiologic statistical models and extensive medical record review to detect safety-related events that are potentially missed by these indicators.
5.
6. 7. 8. 9. 10.
11. 12. 13.
Author Contributions Study conception and design: Kaafarani, Borzecki, Itani, Rosen Acquisition of data: Kaafarani, Borzecki, Loveland, Mull, Hickson, MacDonald, Shin, Rosen Analysis and interpretation of data: Kaafarani, Borzecki, Itani, Loveland, Mull, Shin, Rosen Drafting of manuscript: Kaafarani, Borzecki, Itani, Loveland, Mull, Hickson, MacDonald, Shin, Rosen Critical revision: Kaafarani, Borzecki, Itani, Rosen
14.
15.
16. 17.
REFERENCES 1. Institute of Medicine. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy; 2000. 2. Institute of Medicine. Crossing the Quality Chasm. Washington, DC: National Academy; 2001. 3. Zhan C, Miller MR. Excess length of stay, charges, and mortality attributable to medical injuries during hospitalization. JAMA 2003;290:1868–1874. 4. Miller DC, Filson CP, Wallner LP, et al. Comparing performance of Morbidity and Mortality Conference and National
18.
19.
Validity of Selected Patient Safety Indicators
933
Surgical Quality Improvement Program for detection of complications after urologic surgery. Urology 2006;68:931– 937. Hutter MM, Rowell KS, Devaney LA, et al. Identification of surgical complications and deaths: an assessment of the traditional surgical morbidity and mortality conference compared with the American College of Surgeons-National Surgical Quality Improvement Program. J Am Coll Surg 2006;203:618– 624. Romano PS, Geppert JJ, Davies S, et al. A national profile of patient safety in U.S. hospitals. Health Aff (Millwood) 2003;22: 154–166. AHRQ Quality Indicators: Guide to Patient Safety Indicators. Version 3.0a. Rockville, MD: Agency for Healthcare Research and Quality (AHRQ); March 2003; version 3.1. Healthgrades patient safety methodology. Available at http:// www.healthgrades.com/media/DMS/pdf/PatientSafety Methodology.pdf. Accessed April 15, 2010. National Quality Forum. NQF endorsed standards. Available at: http://www.qualityforum.org/Measures_List.aspx. Accessed April 15th, 2010. Centers for Medicare and Medicaid Services. Details for: Quality measures for reporting in fiscal year 2009 for 2010 update. Available at: http://www.cms.hhs.gov/apps/media/press/factsheet. asp?Counter⫽3044&intNumPerPage⫽10&checkDate⫽& checkKey⫽&srchType⫽1&numDays⫽3500&srchOpt⫽ 0&srchData⫽&keywordType⫽All&chkNewsType⫽6& intPage⫽&showAll⫽&pYear⫽&year⫽&desc⫽false&cbo Order⫽date. Accessed April 15, 2010. QualityNet. Available at: http://www.qualitynet.org. Accessed April 15, 2010. Bahl V, Thompson MA, Kau TY, et al. Do the AHRQ patient safety indicators flag conditions that are present at the time of hospital admission? Med Care 2008;46:516–522. Houchens RL, Elixhauser A, Romano PS. How often are potential patient safety events present on admission? Jt Comm J Qual Patient Saf 2008;34:154–163. White RH, Sadeghi B, Tancredi DJ, et al. How valid is the ICD-9-CM based AHRQ patient safety indicator for postoperative venous thromboembolism? Med Care 2009;47:1237– 1243. Henderson KE, Recktenwald A, Reichley RM, et al. Clinical validation of the AHRQ postoperative venous thromboembolism patient safety indicator. Jt Comm J Qual Patient Saf 2009; 35:370–376. Utter GH, Zrelak PA, Baron R, et al. Positive predictive value of the AHRQ accidental puncture or laceration patient safety indicator. Ann Surg 2009;250:1041–1045. Weller WE, Gallagher BK, Cen L, Hannan EL. Readmissions for venous thromboembolism: expanding the definition of patient safety indicators. Jt Comm J Qual Saf 2004;30:497– 504. Gallagher B, Cen L, Hannan EL. Validation of AHRQ’s patient safety indicator for accidental puncture or laceration. Available at: http://www.ncbi.nlm.nih.gov/bookshelf/br.fcgi?book⫽ aps2&part⫽A1586. Accessed April 15, 2010. Zhan C, Battles J, Chiang YP, Hunt D. The validity of ICD9-CM codes in identifying postoperative deep vein thrombosis and pulmonary embolism. Jt Comm J Qual Patient Saf 2007; 33:326–331.
934
Kaafarani et al
Validity of Selected Patient Safety Indicators
20. Kaafarani HM, Rosen AK. Using administrative data to identify surgical adverse events: an introduction to the Patient Safety Indicators. Am J Surg 2009;198:S63–68. 21. Quality AfHRa. Patient Safety Indicators: technical specifications, March 2003, version 3.1. 22. Sadeghi B, Baron R, Zrelak P, et al. Cases of iatrogenic pneumothorax can be identified from ICD-9-CM coded data. Am J Med Qual 2010;25:218–224.
J Am Coll Surg
23. Allison JJ, Wall TC, Spettell CM, et al. The art and science of chart review. Jt Comm J Qual Improv 2000;26:115–136. 24. Kashner TM. Agreement between administrative files and written medical records: a case of the Department of Veterans Affairs. Med Care 1998;36:1324–1336. 25. Petersen LA, Wright S, Normand SL, Daley J. Positive predictive value of the diagnosis of acute myocardial infarction in an administrative database. J Gen Intern Med 1999;14:555–558.