- Email: [email protected]
Re: Practice Guidelines for Imaging Studies in Children After the First Urinary Tract Infection
2134
LETTERS TO THE EDITOR/ERRATA
Respectfully, Paul Cathcart, Stephen Connolly, Thomas Walton, Anthony J. Costello and Declan G. Murphy Department of Urology, Royal Melbourne Hospital Australian Prostate Cancer Research Centre, Epworth Richmond Hospital and Division of Cancer Surgery, Peter MacCallum Cancer Center Melbourne, Australia e-mail: [email protected] 1. Hu JC, Gu X, Lipsitz SR et al: Comparative effectiveness of minimally invasive vs open radical prostatectomy. JAMA 2009; 302: 1557.
databases, to prospective risk-adjusted analysis— how far have we come? Curr Opin Pediatr 2008; 20: 320.
2. Morrato EH, Dillon P and Ziegler M: Surgical outcomes research: a progression from performance audits, to assessment of administrative
3. Peat J and Barton B: Medical Statistics: A Guide to Data Analysis and Critical Appraisal. Boston: Blackwell Publishing 2005.
Reply by Authors: We thank Cathcart et al for their thoughtful comments and interest in our article. Fundamentally we agree with many of their concerns. As we stated in the article, this is a retrospective, single institutional study that largely reflects our institutional practice of coding postoperative complications. Ideally this study would serve as an impetus to examine these outcomes in a multi-institutional fashion. Yet differences in the collection of self-questionnaire data (if it is performed at all) make a multi-institutional study of such complications difficult. We also agree that the specificity for identifying urinary incontinence exceeds 0.9, yielding a high accuracy. However, the use of accuracy, and even specificity, may not be as practical when the 2 classes are of exceedingly different sizes (using this definition, 98.9% of patients were classified as continent). For example assigning every patient to the continent group would achieve a high proportion of correct predictions (accuracy) but would not provide useful distinction. In the end this study does not diminish the many useful applications of administrative databases in surgical outcomes research. However, the study does provide pause that such databases are not designed to answer every question. Given the difficulties in defining UI and ED even with dedicated instruments, it is disappointing, yet not surprising, that administrative databases failed to correlate with such poorly defined outcomes.
Re: Practice Guidelines for Imaging Studies in Children After the First Urinary Tract Infection M. Venhola, N. P. Huttunen, M. Renko, T. Pokka and M. Uhari J Urol 2010; 184: 325–328.
To the Editor: This 2 center, retrospective, cohort study aims to validate the decision rule derived by Oostenbrink et al,1 and includes a much higher number of patients than the derivation study itself. The authors state that the decision rule is of poor interest, as it resulted in 24% sensitivity instead of the 100% sensitivity found by Oostenbrink et al. They then naturally conclude that the rule is not reproducible, as we stated in 2006 in a smaller validation cohort study.2 A key point when performing a validation study, especially when the reproducibility fails, is to be sure that the lack of validation can be related to the rule itself.3 Indeed, a rule can be nonreproducible because of too wide a difference between derivation and validation populations, explaining the failure of transportability, or because of limitations of the validation study. These 2 options should be considered and explored before concluding maybe too quickly a lack of interest regarding the rule.3 In the present case we have some concern with the validation study. The authors reported not having included in the calculation of the individual risk score the item “family history” because it was not available from their data. However, this item was heavily weighted (coefficient ⫹5) in the logistic regression equation, which surely underestimated the risk score for some patients, leading to the observed decrease of sensitivity. Furthermore, the flow of patients who underwent radiological or radionuclide cystography or renal ultrasound appeared to us sometimes confusing, and it remained unclear for exactly how many patients a risk score would have been calculated. Lastly it would be of value if the authors provided confidence intervals of sensitivity and specificity to give a more precise
LETTERS TO THE EDITOR/ERRATA
2135
idea of the lack of reproducibility, and we would be interested to read the comparison between validation and derivation populations regarding the main baseline characteristics. We agree that all of these considerations may appear technical, and up to a point boring or uninteresting. However, a validation study is at least as important as the derivation study in the whole picture of the construction of a clinical decision tool that aims to help clinicians make better and evidence-based medicine decisions for the benefit of the patient.4 Concluding too quickly that a decision rule is not of interest, based on a validation study for which limitations are ignored, is a shame for the research process, although in the present case we agree with the authors that the rule had real methodological weaknesses.2 We hope that clinical research will aim for and reach high quality standards to be able to state robust conclusions that will benefit patient care. Respectfully, Sandrine Leroy Epidemiology of Emerging Diseases Unit Institut Pasteur Paris and
Pierre Cochat Service de Néphrologie et Rhumatologie Centre de Référence des Maladies Rénales Rares ‘Néphrogones’ Hôpital Femme Mère Enfant and Université de Lyon Bron, France
Reply by Authors: We thank Leroy and Cochat for their interest in our study, and for the important questions they present. We consider the populations in the study of Oostenbrink et al and our study to be quite similar,1 since the age of the children, the type of urinary tract infection and most importantly the vesicoureteral reflux (VUR) grades were the same in both populations. Thus, the difference in our results is not due to the difference in study groups. What is important is that in our study population there were 69 children (20%) with grade III to V reflux, and this is just the patient group that we should be able to identify with the clinical decision rule. The modest sensitivity of this rule (24%) in our validation study in identifying these VUR cases makes it impractical in clinical use. Since Oostenbrink et al do not define what “positive family history” comprises, it is impossible to include this item in a validation study of their decision rule. Since they do not define family, or whether family members should be examined for VUR systematically or randomly based on symptoms, it is impossible to reproduce this item. All patients in our study underwent voiding cystography, and all were also included in the calculation of the risk score. Because the sensitivity of the prediction model of Oostenbrink et al in our study population appeared to be small, even wide confidence intervals of sensitivity would not have a marked influence on the interpretation of our study, although we agree that it would have been important to give the confidence intervals, especially since our result was negative. We have now calculated the 95% confidence intervals, and they vary from 14 to 38, showing that the decision rule does not work even at its best performance. 1. Oostenbrink R, van der Heijden AJ, Moons KG et al: Prediction of vesico-ureteric reflux in childhood urinary tract infection: a multivariate approach. Acta Paediatr 2000; 89: 806.
3. Altman DG, Vergouwe Y, Royston P et al: Prognosis and prognostic research: validating a prognostic model. BMJ 2009; 338: b605.
2. Leroy S, Marc E, Adamsbaum C et al: Prediction of vesicoureteral reflux after a first febrile urinary tract infection in children: validation of a clinical decision rule. Arch Dis Child 2006; 91: 241.
4. Laupacis A, Sekar N and Stiell IG: Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 1997; 277: 488.
ERRATUM PROGNOSTIC PARAMETERS IN NONSEMINOMATOUS GERM CELL TESTICULAR TUMORS Volume 186, Number 4, Part I, Page 1301: In line 8 Cohen’s ⫽ 0.57 as in table 2.
LETTERS TO THE EDITOR/ERRATA
Respectfully, Paul Cathcart, Stephen Connolly, Thomas Walton, Anthony J. Costello and Declan G. Murphy Department of Urology, Royal Melbourne Hospital Australian Prostate Cancer Research Centre, Epworth Richmond Hospital and Division of Cancer Surgery, Peter MacCallum Cancer Center Melbourne, Australia e-mail: [email protected] 1. Hu JC, Gu X, Lipsitz SR et al: Comparative effectiveness of minimally invasive vs open radical prostatectomy. JAMA 2009; 302: 1557.
databases, to prospective risk-adjusted analysis— how far have we come? Curr Opin Pediatr 2008; 20: 320.
2. Morrato EH, Dillon P and Ziegler M: Surgical outcomes research: a progression from performance audits, to assessment of administrative
3. Peat J and Barton B: Medical Statistics: A Guide to Data Analysis and Critical Appraisal. Boston: Blackwell Publishing 2005.
Reply by Authors: We thank Cathcart et al for their thoughtful comments and interest in our article. Fundamentally we agree with many of their concerns. As we stated in the article, this is a retrospective, single institutional study that largely reflects our institutional practice of coding postoperative complications. Ideally this study would serve as an impetus to examine these outcomes in a multi-institutional fashion. Yet differences in the collection of self-questionnaire data (if it is performed at all) make a multi-institutional study of such complications difficult. We also agree that the specificity for identifying urinary incontinence exceeds 0.9, yielding a high accuracy. However, the use of accuracy, and even specificity, may not be as practical when the 2 classes are of exceedingly different sizes (using this definition, 98.9% of patients were classified as continent). For example assigning every patient to the continent group would achieve a high proportion of correct predictions (accuracy) but would not provide useful distinction. In the end this study does not diminish the many useful applications of administrative databases in surgical outcomes research. However, the study does provide pause that such databases are not designed to answer every question. Given the difficulties in defining UI and ED even with dedicated instruments, it is disappointing, yet not surprising, that administrative databases failed to correlate with such poorly defined outcomes.
Re: Practice Guidelines for Imaging Studies in Children After the First Urinary Tract Infection M. Venhola, N. P. Huttunen, M. Renko, T. Pokka and M. Uhari J Urol 2010; 184: 325–328.
To the Editor: This 2 center, retrospective, cohort study aims to validate the decision rule derived by Oostenbrink et al,1 and includes a much higher number of patients than the derivation study itself. The authors state that the decision rule is of poor interest, as it resulted in 24% sensitivity instead of the 100% sensitivity found by Oostenbrink et al. They then naturally conclude that the rule is not reproducible, as we stated in 2006 in a smaller validation cohort study.2 A key point when performing a validation study, especially when the reproducibility fails, is to be sure that the lack of validation can be related to the rule itself.3 Indeed, a rule can be nonreproducible because of too wide a difference between derivation and validation populations, explaining the failure of transportability, or because of limitations of the validation study. These 2 options should be considered and explored before concluding maybe too quickly a lack of interest regarding the rule.3 In the present case we have some concern with the validation study. The authors reported not having included in the calculation of the individual risk score the item “family history” because it was not available from their data. However, this item was heavily weighted (coefficient ⫹5) in the logistic regression equation, which surely underestimated the risk score for some patients, leading to the observed decrease of sensitivity. Furthermore, the flow of patients who underwent radiological or radionuclide cystography or renal ultrasound appeared to us sometimes confusing, and it remained unclear for exactly how many patients a risk score would have been calculated. Lastly it would be of value if the authors provided confidence intervals of sensitivity and specificity to give a more precise
LETTERS TO THE EDITOR/ERRATA
2135
idea of the lack of reproducibility, and we would be interested to read the comparison between validation and derivation populations regarding the main baseline characteristics. We agree that all of these considerations may appear technical, and up to a point boring or uninteresting. However, a validation study is at least as important as the derivation study in the whole picture of the construction of a clinical decision tool that aims to help clinicians make better and evidence-based medicine decisions for the benefit of the patient.4 Concluding too quickly that a decision rule is not of interest, based on a validation study for which limitations are ignored, is a shame for the research process, although in the present case we agree with the authors that the rule had real methodological weaknesses.2 We hope that clinical research will aim for and reach high quality standards to be able to state robust conclusions that will benefit patient care. Respectfully, Sandrine Leroy Epidemiology of Emerging Diseases Unit Institut Pasteur Paris and
Pierre Cochat Service de Néphrologie et Rhumatologie Centre de Référence des Maladies Rénales Rares ‘Néphrogones’ Hôpital Femme Mère Enfant and Université de Lyon Bron, France
Reply by Authors: We thank Leroy and Cochat for their interest in our study, and for the important questions they present. We consider the populations in the study of Oostenbrink et al and our study to be quite similar,1 since the age of the children, the type of urinary tract infection and most importantly the vesicoureteral reflux (VUR) grades were the same in both populations. Thus, the difference in our results is not due to the difference in study groups. What is important is that in our study population there were 69 children (20%) with grade III to V reflux, and this is just the patient group that we should be able to identify with the clinical decision rule. The modest sensitivity of this rule (24%) in our validation study in identifying these VUR cases makes it impractical in clinical use. Since Oostenbrink et al do not define what “positive family history” comprises, it is impossible to include this item in a validation study of their decision rule. Since they do not define family, or whether family members should be examined for VUR systematically or randomly based on symptoms, it is impossible to reproduce this item. All patients in our study underwent voiding cystography, and all were also included in the calculation of the risk score. Because the sensitivity of the prediction model of Oostenbrink et al in our study population appeared to be small, even wide confidence intervals of sensitivity would not have a marked influence on the interpretation of our study, although we agree that it would have been important to give the confidence intervals, especially since our result was negative. We have now calculated the 95% confidence intervals, and they vary from 14 to 38, showing that the decision rule does not work even at its best performance. 1. Oostenbrink R, van der Heijden AJ, Moons KG et al: Prediction of vesico-ureteric reflux in childhood urinary tract infection: a multivariate approach. Acta Paediatr 2000; 89: 806.
3. Altman DG, Vergouwe Y, Royston P et al: Prognosis and prognostic research: validating a prognostic model. BMJ 2009; 338: b605.
2. Leroy S, Marc E, Adamsbaum C et al: Prediction of vesicoureteral reflux after a first febrile urinary tract infection in children: validation of a clinical decision rule. Arch Dis Child 2006; 91: 241.
4. Laupacis A, Sekar N and Stiell IG: Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 1997; 277: 488.
ERRATUM PROGNOSTIC PARAMETERS IN NONSEMINOMATOUS GERM CELL TESTICULAR TUMORS Volume 186, Number 4, Part I, Page 1301: In line 8 Cohen’s ⫽ 0.57 as in table 2.