- Email: [email protected]
Assessing the Interrater Variability of the Mortality Probability Model III
October 2010, Vol 138, No. 4_MeetingAbstracts Poster Presentations: Tuesday, November 2, 2010 | October 2010
Assessing the Interrater Variability of the Mortality Probability Model III Chee M. Chan, MD; Carren Wang, MD; Phuong D. Ho, MD; Andrew F. Shorr, MD Washington Hospital Center, Washington, DC Chest. 2010;138(4_MeetingAbstracts):290A. doi:10.1378/chest.9906 Abstract PURPOSE: Severity of illness scoring is often utilized for benchmarking and quality assessment purposes. The Mortality Probability Model (MPM) III represents one such tool that is easy to calculate. However, the inter-rater variability of the MPM III score is unknown. METHODS: Two clinicians retrospectively calculated the MPM III score in select patients admitted to our Intermediate Care Unit. The scorers computed the MPM III values blinded both to the calculations of the other scorer and to the patient's eventual outcome. After calculation of raw scores, patients were categorized as to either low (010%), intermediate (11-20%) and high (>20%) predicted risk for hospital death as indicated by the scores. We further dichotomized persons into a high risk group (>20% expected mortality) vs a low risk (< 20% expected mortality). We determined the intraclass correlation to measure inter-rater variability for the raw MPM III score and utilized kappa statistics for evaluating inter-rater agreement for the other analyses. RESULTS: The cohort consisted of 152 subjects (mean age: 60.0+/- 14.9; male: 42.1%, 3.9% hospital mortality). The mean MPM III score was 14.0 +/- 12.7 for observer 1 and 3.8 +/- 3.2 for observer 2. The intraclass correlation of the MPM III was poor (0.311; 95% CI: 0.08-0.55) and worsened after segregation into risk groups. The kappa statistic was 0.053 (p=0.066) based on the 3 possible mortality groups and 0.099 (p=0.005) when scores were dichotomized to high vs low risk for death. The observers disagreed in 43% of cases as to whether the patient had a > 20% vs. <20% risk for death. CONCLUSION: The inter-rater agreement of the MPM III is poor. Dichotomizing the MPM score into various strata of risk for death does not result in improved agreement. CLINICAL IMPLICATIONS: The high degree of inter-rater disagreement suggests that the MPM III score should not be employed for purposes of benchmarking or quality assurance. Future efforts should focus on developing risk stratification tools that are both accurate and minimize inter-rater variability. DISCLOSURE: Chee Chan, No Financial Disclosure Information; No Product/Research Disclosure Information 12:45 PM - 2:00 PM
Assessing the Interrater Variability of the Mortality Probability Model III Chee M. Chan, MD; Carren Wang, MD; Phuong D. Ho, MD; Andrew F. Shorr, MD Washington Hospital Center, Washington, DC Chest. 2010;138(4_MeetingAbstracts):290A. doi:10.1378/chest.9906 Abstract PURPOSE: Severity of illness scoring is often utilized for benchmarking and quality assessment purposes. The Mortality Probability Model (MPM) III represents one such tool that is easy to calculate. However, the inter-rater variability of the MPM III score is unknown. METHODS: Two clinicians retrospectively calculated the MPM III score in select patients admitted to our Intermediate Care Unit. The scorers computed the MPM III values blinded both to the calculations of the other scorer and to the patient's eventual outcome. After calculation of raw scores, patients were categorized as to either low (010%), intermediate (11-20%) and high (>20%) predicted risk for hospital death as indicated by the scores. We further dichotomized persons into a high risk group (>20% expected mortality) vs a low risk (< 20% expected mortality). We determined the intraclass correlation to measure inter-rater variability for the raw MPM III score and utilized kappa statistics for evaluating inter-rater agreement for the other analyses. RESULTS: The cohort consisted of 152 subjects (mean age: 60.0+/- 14.9; male: 42.1%, 3.9% hospital mortality). The mean MPM III score was 14.0 +/- 12.7 for observer 1 and 3.8 +/- 3.2 for observer 2. The intraclass correlation of the MPM III was poor (0.311; 95% CI: 0.08-0.55) and worsened after segregation into risk groups. The kappa statistic was 0.053 (p=0.066) based on the 3 possible mortality groups and 0.099 (p=0.005) when scores were dichotomized to high vs low risk for death. The observers disagreed in 43% of cases as to whether the patient had a > 20% vs. <20% risk for death. CONCLUSION: The inter-rater agreement of the MPM III is poor. Dichotomizing the MPM score into various strata of risk for death does not result in improved agreement. CLINICAL IMPLICATIONS: The high degree of inter-rater disagreement suggests that the MPM III score should not be employed for purposes of benchmarking or quality assurance. Future efforts should focus on developing risk stratification tools that are both accurate and minimize inter-rater variability. DISCLOSURE: Chee Chan, No Financial Disclosure Information; No Product/Research Disclosure Information 12:45 PM - 2:00 PM