- Email: [email protected]
The CRIB score
Letters to the Editor
The CRIB
score
SIR-At first reading of the International Neonatal Network’s report (July 24, p 193), I felt encouraged that here at last was evidence that neonatal intensive care units in district hospitals needed more staff and equipment. However, closer scrutiny of the paper reveals difficulties with the study. Teaching hospitals and district hospitals are not defined. Definition of a teaching hospital can vary according to criteria of staffing and equipment. Were hospitals classified retrospectively, perhaps to reinforce the preconceived ideas of the authors? About 40 variables were included in the multivariate analysis to construct the statistical model that constitutes the CRIB score. It is surprising that pre-eclampsia and use of antenatal steroids are not significantly associated with neonatal mortality, since this finding conflicts with evidence from other studies.1 The 40 variables were presumably identified from only neonatal case notes, and the transcription from maternal notes of important obstetric complications and treatments may have been incomplete. Other important possible prognostic factors seem to have been omitted, such as antepartum haemorrhage, chorioamnionitis, prolonged rupture of the membranes, and intrapartum fetal hypoxia. These variables, added to the logistic model, may have altered substantially the magnitude of the regression coefficients of the other significant variables and thus also the integers used for the scoring system. The list of explanatory variables includes measures of neonatal hypoxia and acidosis; this is wrong, because these are response, not explanatory, variables. Thus the same factors that result in neonatal death (prematurity, birth asphyxia, fetal infection) will also result in neonatal hypoxia and acidosis. In figure 3 neonatal mortality of each of the neonatal units is compared with the best one. No allowance is made for these multiple comparisons.2 A nominal level of statistical significance closer to 0-01 than to 0 05 should have been set, to allow for the possibility of a statistically significant difference occurring by chance. The use of this higher probability value might have resulted in the differences between the standard unit and the district hospitals being no longer significant. With respect to normally formed infants, birthweight might be the only reasonable factor that will predict neonatal outcome. This possibility is suggested by figure 2, which shows that the curves for CRIB score and birthweight are close together: at 95% specificity, one poor predictor of neonatal mortality, the CRIB score, with a sensitivity of 51 %, is only slightly better than another poor predictor of neonatal mortality, birthweight, with a sensitivity of about 40%. The information in the CRIB investigator’s report cannot therefore be used as evidence of worse performance in district hospitals. This study illustrates the difficulties of identifying the relative importance of factors that are associated with neonatal mortality, and of then constructing a scoring system that will predict neonatal mortality and that can be used to compare different neonatal units.
1
2
Crowley P, Chalmers I, Keirse MJNC. The effects of corticosteroid administration before preterm delivery: an overview of the evidence from controlled trials. Br J Obstet Gynaecol 1990; 97: 11-25. Armitage P, Berry G. Statistical methods in medical research, 2nd ed. Oxford: Blackwell Scientific Publications, 1987: 444-45.
SiR-The report of the CRIB score is an important step in the performance of individual neonatal units. We raise a note of caution, however, about the distribution of birthweight and gestation, both being major determinants of outcome. We have compared a geographically defined UK population (Trent Health Region) with one from North America (British Columbia, Canada). Inclusion criteria were similar to those described in the CRIB system (32 weeks’ gestation, 1500 g birthweight). We were surprised to find that the Canadian cohort were significantly less mature but with greater birthweights than the Trent cohort. In addition, analysing trends in gestation and birthweight over 4 years has revealed significant changes in these indices within the Trent preterm population. This change in case-mix within a defined preterm population may lead to a false impression of performance of neonatal units over time, as judged by CRIB score. Since the CRIB score was validated on a UK cohort, it may also be important to consider such demographic changes before extrapolating the use of such scores to other countries or ethnic groups.
clarifying
Alan C Fenton, David J Field Department of Child Health, University of Leicester School of Medicine, Leicester Royal Infirmary, Leicester LE2 7LX, UK
Alfonso Solimano, Gail Annich Department of Pediatrics, University of British Columbia, Canada
SiR-The International Neonatal Network has provided paediatricians with a neonatal responsibility with a most useful tool, and I hope that all paediatricians will give urgent consideration to making use of this score and thereby facilitating inter-unit comparisons. How irritating, then, that the investigators interpret their findings in a most deleterious way to non-tertiary hospital units. As they point out, "there was a fourfold increase in odds of death in the non-tertiary hospital neonatal unit with the highest risk-adjusted mortality compared with the tertiary hospital with the lowest risk-adjusted mortality by CRIB". Figure 3 (rankings of hospital performance) illustrates this statement well for the reader. Figure 3 also seems to illustrate, however, a possible alternative description of the hospitals studied, by use of the CRIB score, placing the hospitals in three groups. This description is of four top ranking hospitals, the index tertiary hospital 1, and tertiary hospitals 3, 6, 7; six next ranking, these being tertiary hospitals 4, 2, 8, 5, 9, and non-tertiary hospital A, with CRIB-adjusted risk of death twice that of the index hospital; and three worst ranking, non-tertiary C, B, D, with CRIB-adjusted risk of death four times that of the index
hospital. John M Grant Bellshill Maternity Hospital, Bellshill, Lanarkshire ML4 3JN, UK
612
"Five tertiary hospitals have death rates twice that of the best"; how does that headline compare with "worst non-
tertiary hospital hospital"?
ranks four times
worse
than best ,tertiary
Peter Ehrhardt Burnley General Hospital, Burnley BB10 2PQ, UK
However, thre is growing evidence that initial disease severity is at least as important a prognostic variable as birthweight and gestation (CRIB report and refs 4,5), so perhaps it should be
routinely recorded in geographically defined studies. We appreciate Ehrhardt’s endorsement of the CRIB
Authors’reply SiR-With reference to Grant’s queries: we prospectively defined teaching hospitals as those where medical students received most of their clinical training. The research assistants examined the maternal notes of each infant. On univariate analysis in the development cohort, pre-eclampsia was not associated with death, perhaps because if small numbers rather than lack of a real effect, but use of antenatal steroids was strongly associate with survival, as expected.’ On multiple regression analysis, however, antenatal steroids were no longer independently related to survival after adjusting for CRIB
(table). Antenatal steroids seemed, therefore, to exert their effects by reducing initial disease severity, so infants whose mothers were untreated had higher CRIB scores and vice versa. Prolonged rupture of the membranes was only recorded in the validation cohort, and neither it nor antepartum haemorrhage were related to mortality on univariate analysis. Chorioamnionitis and intrapartum fetal hypoxia were not recorded because of doubts about uniformity of definitions in a retrospective study. In a logistic model with death as the dependent variable, oxygenation and worst (most acidotic) base excess in the first 12 h were strongly independently associated with death-ie, were major explanatory variables that improved CRIB’s predictive power, as in similar scores (a normal procedure in logistic regression analysis).2 Our main aim was to make a risk-adjuted comparison of mortality between the two groups of hospitals, for which a Bonferroni correction is not appropriate. Our conclusion that there was a twofold difference in odds of death between the two groups thus remains unchanged. Table 2 and figure 2 provide no support for the claim that in normally formed infants "birthweight might be the only reasonable factor that will predict neonatal outcome". In fact they show the opposite-that CRIB is a better predictor than birthweight. Table 4 shows that CRIB is only slightly less accurate than scores needing considerably more data. A score with no predictive value would predict death with only 4% sensitivity at a specificity of 96%. The widely used PRISM score, which predicted death with 51% sensitivity at 96% specificity, revealed big disparities in risk-adjusted mortality between tertiary and non-tertiary paediatric intensive care units.3 To insist on near-perfection before applying such a useful tool is a recipe for inertia that might delay improvements in care. Grant notes the lack of evidence that neonatal intensive care units in district general hospitals require more staff and equipment. One remedy would be to find out if hospitals that have less resources perform less well. The CRIB or SNAP4 scores provide valid tools for this task. We agree with Fenton and colleagues’ implicit proposal that CRIB should be further developed and validated in international cohorts and that interactions between birthweight, gestation, or ethnic group be investigated.
score
have irritated him. The fourfold difference in risk-adjusted odds of death at the extremes of the rankings was reported not to degingrate one type of hospital but to show how scores that use shorter periods to sample initial disease severity, such as CRIB, might reveal greater disparities in performance than scores that sample over 24 h. Although, as Ehrhardt points out, the risk-adjusted odds of death in some teaching hospitals were twice those in the reference centre, the confidence intervals were wide and the differences were not significant, which shows the difficulties of interpreting league tables. In fact our interpretation was that fine ranking of individual hospitals in studies of this size should be avoided and reliable comparisons can only be made between groups of hospitals. Also, our data from 13 hospitals provide no insight into the picture in the UK as a whole For this definitive national study is needed. We believe that it is now appropriate to develop and apply CRIB and SNAP further in prospective, longitudinal, nationally representative, or geographically based studies of risk-adjusted performance of neonatal units in relation to staffing, equipment, policy, and organisation. Although few randomised controlled trials of individual therapies show differences in mortality of more than 20-40%, risk adjusted comparisons between intensive care units have shown differences ten times larger (CRIB study and refs 5-7). National and international studies could improve our understanding of how best to organise neonatal, and perhaps paediatric and adult, intensive care.’ The International Neonatal Network and the British Association of Perinatal Medicine therefore invite all neonatal paediatricians, both in the UK and elsewhere, to join a large collaborative study to refine and apply neonatal scoring systems more widely.
but
are
sorry
to
William Tarnow-Mordi, Richard Simon Ogston
Cooke, Gareth Parry,
Centre for Research into Human Development, University of Dundee, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
1
2 3
4
5
6 7
Crowley P, Chalmers I, Keirse MJNC. The effect of corticosteroid administration before preterm delivery: an overview of the evidence from controlled trials. Br J Obstet Gynaecol 1990; 97: 11-25. Pollack MM, Ruttimann UE, Getson PR. Pediatric risk of mortality (PRISM) score. Crit Care Med 1988; 16: 1110-16. Pollack MM, Alexander SR, Clarke N, et al. Improved outcomes from tertiary center pediatric intensive care: a statewide comparison of tertiary and nontertiary care facilities. Crit Care Med 1991; 19: 150-59. Richardson DK, Phibbs CS, Gray JE, et al. Birthweight and illness severity. Independent predictors of NICU mortality. Pediatrics 1993; 91: 969-75. Tarnow-Mordi WO, Ogsten SA, Wilkinson AR, et al. Predicting death from initial disease severity: a method for comparing the performance of neonatal units. BMJ 1990; 300: 1611-14. Field D, Hodges S, Mason E, Burton P. Survival and place of treatment after premature delivery. Arch Dis Child 1991; 66: 408-11. Pollack MM, Cuerdon TC, Getson PR. Pediatric intensive care units: results of a national survey. Crit Care Med 1993; 21: 607-14.
Table: Relation between antenatal steroids and adverse neonatal events
613
The CRIB
score
SIR-At first reading of the International Neonatal Network’s report (July 24, p 193), I felt encouraged that here at last was evidence that neonatal intensive care units in district hospitals needed more staff and equipment. However, closer scrutiny of the paper reveals difficulties with the study. Teaching hospitals and district hospitals are not defined. Definition of a teaching hospital can vary according to criteria of staffing and equipment. Were hospitals classified retrospectively, perhaps to reinforce the preconceived ideas of the authors? About 40 variables were included in the multivariate analysis to construct the statistical model that constitutes the CRIB score. It is surprising that pre-eclampsia and use of antenatal steroids are not significantly associated with neonatal mortality, since this finding conflicts with evidence from other studies.1 The 40 variables were presumably identified from only neonatal case notes, and the transcription from maternal notes of important obstetric complications and treatments may have been incomplete. Other important possible prognostic factors seem to have been omitted, such as antepartum haemorrhage, chorioamnionitis, prolonged rupture of the membranes, and intrapartum fetal hypoxia. These variables, added to the logistic model, may have altered substantially the magnitude of the regression coefficients of the other significant variables and thus also the integers used for the scoring system. The list of explanatory variables includes measures of neonatal hypoxia and acidosis; this is wrong, because these are response, not explanatory, variables. Thus the same factors that result in neonatal death (prematurity, birth asphyxia, fetal infection) will also result in neonatal hypoxia and acidosis. In figure 3 neonatal mortality of each of the neonatal units is compared with the best one. No allowance is made for these multiple comparisons.2 A nominal level of statistical significance closer to 0-01 than to 0 05 should have been set, to allow for the possibility of a statistically significant difference occurring by chance. The use of this higher probability value might have resulted in the differences between the standard unit and the district hospitals being no longer significant. With respect to normally formed infants, birthweight might be the only reasonable factor that will predict neonatal outcome. This possibility is suggested by figure 2, which shows that the curves for CRIB score and birthweight are close together: at 95% specificity, one poor predictor of neonatal mortality, the CRIB score, with a sensitivity of 51 %, is only slightly better than another poor predictor of neonatal mortality, birthweight, with a sensitivity of about 40%. The information in the CRIB investigator’s report cannot therefore be used as evidence of worse performance in district hospitals. This study illustrates the difficulties of identifying the relative importance of factors that are associated with neonatal mortality, and of then constructing a scoring system that will predict neonatal mortality and that can be used to compare different neonatal units.
1
2
Crowley P, Chalmers I, Keirse MJNC. The effects of corticosteroid administration before preterm delivery: an overview of the evidence from controlled trials. Br J Obstet Gynaecol 1990; 97: 11-25. Armitage P, Berry G. Statistical methods in medical research, 2nd ed. Oxford: Blackwell Scientific Publications, 1987: 444-45.
SiR-The report of the CRIB score is an important step in the performance of individual neonatal units. We raise a note of caution, however, about the distribution of birthweight and gestation, both being major determinants of outcome. We have compared a geographically defined UK population (Trent Health Region) with one from North America (British Columbia, Canada). Inclusion criteria were similar to those described in the CRIB system (32 weeks’ gestation, 1500 g birthweight). We were surprised to find that the Canadian cohort were significantly less mature but with greater birthweights than the Trent cohort. In addition, analysing trends in gestation and birthweight over 4 years has revealed significant changes in these indices within the Trent preterm population. This change in case-mix within a defined preterm population may lead to a false impression of performance of neonatal units over time, as judged by CRIB score. Since the CRIB score was validated on a UK cohort, it may also be important to consider such demographic changes before extrapolating the use of such scores to other countries or ethnic groups.
clarifying
Alan C Fenton, David J Field Department of Child Health, University of Leicester School of Medicine, Leicester Royal Infirmary, Leicester LE2 7LX, UK
Alfonso Solimano, Gail Annich Department of Pediatrics, University of British Columbia, Canada
SiR-The International Neonatal Network has provided paediatricians with a neonatal responsibility with a most useful tool, and I hope that all paediatricians will give urgent consideration to making use of this score and thereby facilitating inter-unit comparisons. How irritating, then, that the investigators interpret their findings in a most deleterious way to non-tertiary hospital units. As they point out, "there was a fourfold increase in odds of death in the non-tertiary hospital neonatal unit with the highest risk-adjusted mortality compared with the tertiary hospital with the lowest risk-adjusted mortality by CRIB". Figure 3 (rankings of hospital performance) illustrates this statement well for the reader. Figure 3 also seems to illustrate, however, a possible alternative description of the hospitals studied, by use of the CRIB score, placing the hospitals in three groups. This description is of four top ranking hospitals, the index tertiary hospital 1, and tertiary hospitals 3, 6, 7; six next ranking, these being tertiary hospitals 4, 2, 8, 5, 9, and non-tertiary hospital A, with CRIB-adjusted risk of death twice that of the index hospital; and three worst ranking, non-tertiary C, B, D, with CRIB-adjusted risk of death four times that of the index
hospital. John M Grant Bellshill Maternity Hospital, Bellshill, Lanarkshire ML4 3JN, UK
612
"Five tertiary hospitals have death rates twice that of the best"; how does that headline compare with "worst non-
tertiary hospital hospital"?
ranks four times
worse
than best ,tertiary
Peter Ehrhardt Burnley General Hospital, Burnley BB10 2PQ, UK
However, thre is growing evidence that initial disease severity is at least as important a prognostic variable as birthweight and gestation (CRIB report and refs 4,5), so perhaps it should be
routinely recorded in geographically defined studies. We appreciate Ehrhardt’s endorsement of the CRIB
Authors’reply SiR-With reference to Grant’s queries: we prospectively defined teaching hospitals as those where medical students received most of their clinical training. The research assistants examined the maternal notes of each infant. On univariate analysis in the development cohort, pre-eclampsia was not associated with death, perhaps because if small numbers rather than lack of a real effect, but use of antenatal steroids was strongly associate with survival, as expected.’ On multiple regression analysis, however, antenatal steroids were no longer independently related to survival after adjusting for CRIB
(table). Antenatal steroids seemed, therefore, to exert their effects by reducing initial disease severity, so infants whose mothers were untreated had higher CRIB scores and vice versa. Prolonged rupture of the membranes was only recorded in the validation cohort, and neither it nor antepartum haemorrhage were related to mortality on univariate analysis. Chorioamnionitis and intrapartum fetal hypoxia were not recorded because of doubts about uniformity of definitions in a retrospective study. In a logistic model with death as the dependent variable, oxygenation and worst (most acidotic) base excess in the first 12 h were strongly independently associated with death-ie, were major explanatory variables that improved CRIB’s predictive power, as in similar scores (a normal procedure in logistic regression analysis).2 Our main aim was to make a risk-adjuted comparison of mortality between the two groups of hospitals, for which a Bonferroni correction is not appropriate. Our conclusion that there was a twofold difference in odds of death between the two groups thus remains unchanged. Table 2 and figure 2 provide no support for the claim that in normally formed infants "birthweight might be the only reasonable factor that will predict neonatal outcome". In fact they show the opposite-that CRIB is a better predictor than birthweight. Table 4 shows that CRIB is only slightly less accurate than scores needing considerably more data. A score with no predictive value would predict death with only 4% sensitivity at a specificity of 96%. The widely used PRISM score, which predicted death with 51% sensitivity at 96% specificity, revealed big disparities in risk-adjusted mortality between tertiary and non-tertiary paediatric intensive care units.3 To insist on near-perfection before applying such a useful tool is a recipe for inertia that might delay improvements in care. Grant notes the lack of evidence that neonatal intensive care units in district general hospitals require more staff and equipment. One remedy would be to find out if hospitals that have less resources perform less well. The CRIB or SNAP4 scores provide valid tools for this task. We agree with Fenton and colleagues’ implicit proposal that CRIB should be further developed and validated in international cohorts and that interactions between birthweight, gestation, or ethnic group be investigated.
score
have irritated him. The fourfold difference in risk-adjusted odds of death at the extremes of the rankings was reported not to degingrate one type of hospital but to show how scores that use shorter periods to sample initial disease severity, such as CRIB, might reveal greater disparities in performance than scores that sample over 24 h. Although, as Ehrhardt points out, the risk-adjusted odds of death in some teaching hospitals were twice those in the reference centre, the confidence intervals were wide and the differences were not significant, which shows the difficulties of interpreting league tables. In fact our interpretation was that fine ranking of individual hospitals in studies of this size should be avoided and reliable comparisons can only be made between groups of hospitals. Also, our data from 13 hospitals provide no insight into the picture in the UK as a whole For this definitive national study is needed. We believe that it is now appropriate to develop and apply CRIB and SNAP further in prospective, longitudinal, nationally representative, or geographically based studies of risk-adjusted performance of neonatal units in relation to staffing, equipment, policy, and organisation. Although few randomised controlled trials of individual therapies show differences in mortality of more than 20-40%, risk adjusted comparisons between intensive care units have shown differences ten times larger (CRIB study and refs 5-7). National and international studies could improve our understanding of how best to organise neonatal, and perhaps paediatric and adult, intensive care.’ The International Neonatal Network and the British Association of Perinatal Medicine therefore invite all neonatal paediatricians, both in the UK and elsewhere, to join a large collaborative study to refine and apply neonatal scoring systems more widely.
but
are
sorry
to
William Tarnow-Mordi, Richard Simon Ogston
Cooke, Gareth Parry,
Centre for Research into Human Development, University of Dundee, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
1
2 3
4
5
6 7
Crowley P, Chalmers I, Keirse MJNC. The effect of corticosteroid administration before preterm delivery: an overview of the evidence from controlled trials. Br J Obstet Gynaecol 1990; 97: 11-25. Pollack MM, Ruttimann UE, Getson PR. Pediatric risk of mortality (PRISM) score. Crit Care Med 1988; 16: 1110-16. Pollack MM, Alexander SR, Clarke N, et al. Improved outcomes from tertiary center pediatric intensive care: a statewide comparison of tertiary and nontertiary care facilities. Crit Care Med 1991; 19: 150-59. Richardson DK, Phibbs CS, Gray JE, et al. Birthweight and illness severity. Independent predictors of NICU mortality. Pediatrics 1993; 91: 969-75. Tarnow-Mordi WO, Ogsten SA, Wilkinson AR, et al. Predicting death from initial disease severity: a method for comparing the performance of neonatal units. BMJ 1990; 300: 1611-14. Field D, Hodges S, Mason E, Burton P. Survival and place of treatment after premature delivery. Arch Dis Child 1991; 66: 408-11. Pollack MM, Cuerdon TC, Getson PR. Pediatric intensive care units: results of a national survey. Crit Care Med 1993; 21: 607-14.
Table: Relation between antenatal steroids and adverse neonatal events
613