- Email: [email protected]
Author reply: improving prognosis research: standards primary, secondary, and tertiary
Letter to the Editor / Journal of Clinical Epidemiology 60 (2007) 863e866
References [1] Hemingway H. Prognosis research: why is Dr. Lydgate still waiting? J Clin Epi 2006;59:1229e38. [2] McShane LM, Altman DG, Sauerbrei W. Identification of clinically useful cancer prognostic factors: what are we missing? J Natl Cancer Inst 2005;97:1023e5. [3] Altman DG, Lyman GH. Methodological challenges in the evaluation of prognostic factors in breast cancer. Breast Cancer Res Treat 1998;52:289e303. [4] Simon R, Altman DG. Statistical aspects of prognostic factor studies in oncology. Br J Cancer 1994;69:979e85. [5] McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM. Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst 2005;97:1180e4. [6] Riley RD, Abrams KR, Sutton AJ, Lambert PC, Jones DR, Heney D, et al. Reporting of prognostic markers: current problems and development of guidelines for evidence-based practice in the future. Br J Cancer 2003;88:1191e8. [7] Kyzas PA, Loizou KT, Ioannidis JP. Selective reporting biases in cancer prognostic factor studies. J Natl Cancer Inst 2005;97:1043e55. [8] Strengthening the Reporting of Observational Epidemiology (STROBE). Available at. www.strobe-statement.org. [9] Altman DG, De Stavola BL, Love SB, Stepniewska KA. Review of survival analyses published in cancer journals. Br J Cancer 1995; 72:511e8. [10] Burton A, Altman DG. Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines. Br J Cancer 2004;91:4e8. [11] Holla¨nder N, Sauerbrei W. On statistical approaches for the multivariable analysis of prognostic factor studies. In: Auget J-L, Balakrishnan N, Mesbah M, Molenberghs G, editors. Advances in statistical methods for the health sciences. Boston, MA: Birkha¨user; 2006. [12] Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 2006;25: 127e41. [13] Altman DG, Riley RD. An evidence-based approach to prognostic markers. Nature Clin Pract Oncol 2005;2:466e72. [14] Altman DG. Systematic reviews of evaluations of prognostic variables. BMJ 2001;323:224e8. [15] Riley RD, Heney D, Jones DR, Sutton AJ, Lambert PC, Abrams KR, et al. A systematic review of molecular and biological tumor markers in neuroblastoma. Clin Cancer Res 2004;10:4e12. [16] Riley RD, Abrams KR, Lambert PC, Sutton AJ, Altman DG. Where next for evidence synthesis of prognostic marker studies? Improving the quality and reporting of primary studies to facilitate clinically relevant evidence-based results. In: Auget J-L, Balakrishnan N, Mesbah M, Molenberghs G, editors. Advances in statistical methods for the health sciences. Boston, MA: Birkha¨user; 2006. [17] Altman DG, Trivella M, Pezzella F, Harris AL, Pastorino U. Systematic review of multiple studies of prognosis: the feasibility of obtaining individual patient data. I. In: Auget J-L, Balakrishnan N, Mesbah M, Molenberghs G, editors. Advances in statistical methods for the health sciences. Boston, MA: Birkha¨user; 2006. [18] Sauerbrei W, Hollander N, Riley RD, Altman DG. Evidence-based assessment and application of prognostic markers: the long way from single studies to meta-analysis. Commun Stat 2006;35:1333e42. [19] Hayden JA, Cote P, Bombardier C. Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med 2006;144: 427e37. [20] Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. BMJ 2003;326:41e4. [21] Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallelgroup randomized trials. JAMA 2001;285:1987e91.
865
[22] Counsell C, Dennis M. Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis 2001;2:159e70. [23] Windeler J. Prognosisdwhat does the clinician associate with this notion? Stat Med 2000;19:425e30. doi: 10.1016/j.jclinepi.2007.02.004
Author reply: improving prognosis research: standards primary, secondary, and tertiary Riley et al. should be applauded for the Cochrane initiative and for their helpful letter. We agree that what is needed now is action. We have had for some time a working diagnosis of where the problems lie, and their letter illustrates that in large part these insights come from the discipline of biostatistics and the clinical field of cancer. So how do we improve the quality of prognosis research? Here are three specific challenges. (i) Parallel efforts to produce, and implement, standards for primary and secondary research (systematic review) Developing methodological standards is important, but does not go far enough. As Riley et al. point out to some extent standards already exist. An implementation framework is required to ensure that research outputs improve. The imprimatur of Cochrane may be helpful in bringing about change. Focusing solely on methods for systematic review will founder on the quality of the component studies. We may need fewer, better studies, and this will entail a change in culture of clinical research. Some have proposed a moratorium on new studies, whose findings are likely to be false [1]. Many changes are required in order to bring together scientifically cognate, organizationally disparate disciplines. These include the reward structures for individuals and institutions in participating in collaborative research as well as the policies of those who fund, and publish, prognosis research. (ii) Including the ‘‘omician’’ Prognosis research is being revolutionized by ‘‘omicians’’dindividuals working in ‘‘omic’’disciplines including, genomics, proteomics, and metabonomics. Although the standards for the design, conduct, and analysis of randomized trials are relatively stable, and the basic research question is often simple, increasingly neither is true of prognosis research. Currently the information generated by new biological techniques is outstripping our ability to extract robust, generalizable knowledge (or indeed its application). Standards for the analysis of such information are available, but how these are integrated into wider standards in prognosis research requires development. The previous era of prognosis studies was characterized by a relatively small number of simple measures, and statisticians had a leadership role in developing methods of understanding. The next decade might be characterized by biologically
866
Letter to the Editor / Journal of Clinical Epidemiology 60 (2007) 863e866
more complex measures where the leadership in understanding may come jointly from biological, clinical, statistical, and bioinformatic approaches. (iii) Need for field synopsisdwhich research should be prioritized? Prognosis research is often fragmented into subfields with one group tackling a particular set of prognostic factors. For example, those working in biomarkers and those working in psychosocial factors tend to be separate. Systematic reviews usually focus on only one of these subfields. In order to prioritize where new primary research is likely to be most informative, methods are required that synthesize the results of different meta-analyses and primary studies across the field, integrating the whole range of putative prognostic factors and relevant prognostic outcomes. Such field synopses (tertiary level systematic review) might motivate novel, methodologically stronger collaborative research networks.
Currently it seems, we tolerate double standards in clinical science. For randomized trials standards are generally high and clearly enforced among the many organs of academic life. Standards in prognosis research are variable and often low. We need not only to develop quality standards of primary, secondary, and tertiary prognosis research but also to secure their implementation. Harry Hemingway UCL Medical School, Epidemiology and Public Health 1-19 Torrington Place, London WCIEBT, UK E-mail address: [email protected] (H. Hemingway)
Reference [1] Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2:e124. Epub 2005 August 30. doi: 10.1016/j.jclinepi.2007.02.005
References [1] Hemingway H. Prognosis research: why is Dr. Lydgate still waiting? J Clin Epi 2006;59:1229e38. [2] McShane LM, Altman DG, Sauerbrei W. Identification of clinically useful cancer prognostic factors: what are we missing? J Natl Cancer Inst 2005;97:1023e5. [3] Altman DG, Lyman GH. Methodological challenges in the evaluation of prognostic factors in breast cancer. Breast Cancer Res Treat 1998;52:289e303. [4] Simon R, Altman DG. Statistical aspects of prognostic factor studies in oncology. Br J Cancer 1994;69:979e85. [5] McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM. Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst 2005;97:1180e4. [6] Riley RD, Abrams KR, Sutton AJ, Lambert PC, Jones DR, Heney D, et al. Reporting of prognostic markers: current problems and development of guidelines for evidence-based practice in the future. Br J Cancer 2003;88:1191e8. [7] Kyzas PA, Loizou KT, Ioannidis JP. Selective reporting biases in cancer prognostic factor studies. J Natl Cancer Inst 2005;97:1043e55. [8] Strengthening the Reporting of Observational Epidemiology (STROBE). Available at. www.strobe-statement.org. [9] Altman DG, De Stavola BL, Love SB, Stepniewska KA. Review of survival analyses published in cancer journals. Br J Cancer 1995; 72:511e8. [10] Burton A, Altman DG. Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines. Br J Cancer 2004;91:4e8. [11] Holla¨nder N, Sauerbrei W. On statistical approaches for the multivariable analysis of prognostic factor studies. In: Auget J-L, Balakrishnan N, Mesbah M, Molenberghs G, editors. Advances in statistical methods for the health sciences. Boston, MA: Birkha¨user; 2006. [12] Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 2006;25: 127e41. [13] Altman DG, Riley RD. An evidence-based approach to prognostic markers. Nature Clin Pract Oncol 2005;2:466e72. [14] Altman DG. Systematic reviews of evaluations of prognostic variables. BMJ 2001;323:224e8. [15] Riley RD, Heney D, Jones DR, Sutton AJ, Lambert PC, Abrams KR, et al. A systematic review of molecular and biological tumor markers in neuroblastoma. Clin Cancer Res 2004;10:4e12. [16] Riley RD, Abrams KR, Lambert PC, Sutton AJ, Altman DG. Where next for evidence synthesis of prognostic marker studies? Improving the quality and reporting of primary studies to facilitate clinically relevant evidence-based results. In: Auget J-L, Balakrishnan N, Mesbah M, Molenberghs G, editors. Advances in statistical methods for the health sciences. Boston, MA: Birkha¨user; 2006. [17] Altman DG, Trivella M, Pezzella F, Harris AL, Pastorino U. Systematic review of multiple studies of prognosis: the feasibility of obtaining individual patient data. I. In: Auget J-L, Balakrishnan N, Mesbah M, Molenberghs G, editors. Advances in statistical methods for the health sciences. Boston, MA: Birkha¨user; 2006. [18] Sauerbrei W, Hollander N, Riley RD, Altman DG. Evidence-based assessment and application of prognostic markers: the long way from single studies to meta-analysis. Commun Stat 2006;35:1333e42. [19] Hayden JA, Cote P, Bombardier C. Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med 2006;144: 427e37. [20] Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. BMJ 2003;326:41e4. [21] Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallelgroup randomized trials. JAMA 2001;285:1987e91.
865
[22] Counsell C, Dennis M. Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis 2001;2:159e70. [23] Windeler J. Prognosisdwhat does the clinician associate with this notion? Stat Med 2000;19:425e30. doi: 10.1016/j.jclinepi.2007.02.004
Author reply: improving prognosis research: standards primary, secondary, and tertiary Riley et al. should be applauded for the Cochrane initiative and for their helpful letter. We agree that what is needed now is action. We have had for some time a working diagnosis of where the problems lie, and their letter illustrates that in large part these insights come from the discipline of biostatistics and the clinical field of cancer. So how do we improve the quality of prognosis research? Here are three specific challenges. (i) Parallel efforts to produce, and implement, standards for primary and secondary research (systematic review) Developing methodological standards is important, but does not go far enough. As Riley et al. point out to some extent standards already exist. An implementation framework is required to ensure that research outputs improve. The imprimatur of Cochrane may be helpful in bringing about change. Focusing solely on methods for systematic review will founder on the quality of the component studies. We may need fewer, better studies, and this will entail a change in culture of clinical research. Some have proposed a moratorium on new studies, whose findings are likely to be false [1]. Many changes are required in order to bring together scientifically cognate, organizationally disparate disciplines. These include the reward structures for individuals and institutions in participating in collaborative research as well as the policies of those who fund, and publish, prognosis research. (ii) Including the ‘‘omician’’ Prognosis research is being revolutionized by ‘‘omicians’’dindividuals working in ‘‘omic’’disciplines including, genomics, proteomics, and metabonomics. Although the standards for the design, conduct, and analysis of randomized trials are relatively stable, and the basic research question is often simple, increasingly neither is true of prognosis research. Currently the information generated by new biological techniques is outstripping our ability to extract robust, generalizable knowledge (or indeed its application). Standards for the analysis of such information are available, but how these are integrated into wider standards in prognosis research requires development. The previous era of prognosis studies was characterized by a relatively small number of simple measures, and statisticians had a leadership role in developing methods of understanding. The next decade might be characterized by biologically
866
Letter to the Editor / Journal of Clinical Epidemiology 60 (2007) 863e866
more complex measures where the leadership in understanding may come jointly from biological, clinical, statistical, and bioinformatic approaches. (iii) Need for field synopsisdwhich research should be prioritized? Prognosis research is often fragmented into subfields with one group tackling a particular set of prognostic factors. For example, those working in biomarkers and those working in psychosocial factors tend to be separate. Systematic reviews usually focus on only one of these subfields. In order to prioritize where new primary research is likely to be most informative, methods are required that synthesize the results of different meta-analyses and primary studies across the field, integrating the whole range of putative prognostic factors and relevant prognostic outcomes. Such field synopses (tertiary level systematic review) might motivate novel, methodologically stronger collaborative research networks.
Currently it seems, we tolerate double standards in clinical science. For randomized trials standards are generally high and clearly enforced among the many organs of academic life. Standards in prognosis research are variable and often low. We need not only to develop quality standards of primary, secondary, and tertiary prognosis research but also to secure their implementation. Harry Hemingway UCL Medical School, Epidemiology and Public Health 1-19 Torrington Place, London WCIEBT, UK E-mail address: [email protected] (H. Hemingway)
Reference [1] Ioannidis JP. Why most published research findings are false. PLoS Med 2005;2:e124. Epub 2005 August 30. doi: 10.1016/j.jclinepi.2007.02.005