- Email: [email protected]
Response to Allison et al. ‘Meta-analysis of porcine circovirus type 2 (PCV2) vaccines’
Preventive Veterinary Medicine 119 (2015) 94–95
Contents lists available at ScienceDirect
Preventive Veterinary Medicine journal homepage: www.elsevier.com/locate/prevetmed
Reply to the Letter to the Editor Response to Allison et al. ‘Meta-analysis of porcine circovirus type 2 (PCV2) vaccines’ Thank you for the opportunity to respond to the letter submitted by Dr’s Allison, Taylor and Nitzel about the article “Mixed treatment comparison meta-analysis of porcine circovirus type 2 (PCV2) vaccines used in piglets. da Silva et al. (2014) Preventive Veterinary Medicine 117, 413–424” (da Silva et al., 2014). Among several issues raised, Dr’s Allison, Taylor and Nitzel provide a general opinion about the advantages of having more and better data for meta-analysis. We agree with this and ideally more higher-quality data would be better. This is a universal truth for any statistical analysis not just meta-analysis. Dr’s Allison, Taylor and Nitzel are perhaps suggesting that if faced with the same data they might not have conducted a meta-analysis. Clearly, we did not make such a decision and proceeded with the meta-analysis. Such decisions cannot be viewed as correct or incorrect, but are judgments researchers make and readers’ critique. Dr’s Allison, Taylor and Nitzel also have concerns about the sources of the studies used in the meta-analysis, the size of the studies and the approach to execution. We are pleased that the review is reported in a manner that enabled Dr’s Allison, Taylor and Nitzel to identify potential sources of bias. We strongly encourage assessment of bias when interpreting the primary research, and research synthesis is no exception. Several resources are available to encourage readers unfamiliar with critique of reviews (Shea et al., 2007a,b, 2009) and, more specifically, mixed treatment comparisons meta-analysis (Jansen et al., 2011). Research synthesis, like primary research, is a research endeavor and like primary research, there are potential sources of bias. Dr’s Allison, Taylor and Nitzel have pointed out what they consider to be biases. We are encouraged that the transparent reporting approach has allowed such critique and always encourage any end-user to assess the bias before deciding how to use the information provided. Such assessment of biases is sometimes not possible for reviews because critical information about the conduct of the review is missing such as studies included in the review, reasons for inclusion, exact estimates of outcomes and
DOI of the original article:http://dx.doi.org/10.1016/j.prevetmed. 2015.01.012. http://dx.doi.org/10.1016/j.prevetmed.2015.01.011 0167-5877/© 2015 Elsevier B.V. All rights reserved.
approaches to combining data. Some readers might feel the issues identified by Dr’s Allison, Taylor and Nitzel have minimal effect on the outcome; other readers might feel they are egregious; such debate is an integral part of the scientific process. Regardless we are pleased that our approach to reporting enabled such assessment, and encourage others conducting reviews to report in the same manner. We used the PRISMA statement when preparing our review report (Liberati et al., 2009; Moher et al., 2009). Dr’s Allison, Taylor and Nitzel express concern that use of conference proceedings is a potential source of publication bias “For some vaccines, but not others, sources included non-peer reviewed conference abstracts by company authors. Aside from concerns about the reliability of the experimental work, such material is highly subject to publication bias, with only positive trials presented.” Dr’s Allison, Taylor and Nitzel authors propose the direction of bias will be positive i.e., an over estimate of effect. However, we are unaware of documentation that suggests the direction of bias in veterinary vaccines studies and therefore prefer a transparent and inclusive approach to the review. In human health, conference proceedings are associated with less favorable outcomes (the direction opposite to that proposed by Dr’s Allison, Taylor and Nitzel) and therefore the inclusion of conference proceedings in human health clinical trial reviews is considered an approach to mitigating publication bias (Dundar et al., 2006; Hopewell et al., 2007). The Cochran Handbook of Systematic Reviews discusses the pro and cons of including gray literature and there is no consensus (Higgins and Green, 2011). In the area of swine health, the issue is unfortunately further complicated because there is evidence that peer-review publication is not common for vaccines, with less than 10% of conference proceedings subsequently published as peerreview articles. Therefore exclusion of these studies might simply have swapped one potential criticism for another (Brace et al., 2010). Our approach was to be transparent about the papers included and assess and report the risk of bias in all included studies which is consistent with recommended practices. Again, the decision to use data from conference proceedings is a judgment, and other readers might agree or disagree with the inclusion of such studies. Dr’s Allison, Taylor and Nitzel also express concern about our failure to use p values as decision making tools by saying that “Our concern with the “Highlights” is that they further enhance the impression that meaningful differences between the vaccines were shown when they were not.
Reply to the Letter to the Editor / Preventive Veterinary Medicine 119 (2015) 94–95
. . . without significant p-values for between-vaccine comparisons, no conclusion about between-vaccine differences is justified”. We would propose that estimation of effect size and reporting a precision of estimates is a more informative and appropriate approach to reporting and interpreting the data. We do not propose here to discuss the virtues of significance testing as entire books have been devoted to this topic (Morrison and Henkel, 2006). Our approach to reporting reflects the advice of the Cochran Handbook of Systematic Reviews of Interventions – Review authors are advised not to describe results as “not statistically significant” or “non-significant” (Higgins and Green, 2011). We respectably disagree that the highlights are misleading and believe that within the word limits these are a tempered and thoughtful summary of our opinion of the results. Finally after publication, we were asked to determine why the publication by Fraile et al. (2012) was not identified as relevant. Since publication, we have concluded that this manuscript was found by the search but was not passed from the screening team to the data extraction team, a false negative in the screening process. The authors did not report the number of animals in each group at each farm nor did they report if the measures of precision are standard errors of means (SEM) or standard deviation (SD). Thank you References Brace, S., Taylor, D., O’Connor, A.M., 2010. The quality of reporting and publication status of vaccines trials presented at veterinary conferences from 1988 to 2003. Vaccine 28, 5306–5314. da Silva, N., Carriquiry, A., O’Neill, K., Opriessnig, T., O’Connor, A.M., 2014. Mixed treatment comparison meta-analysis of porcine circovirus type 2 (PCV2) vaccines used in piglets. Prev. Vet. Med. 117, 413–424. Dundar, Y., Dodd, S., Dickson, R., Walley, T., Haycox, A., Williamson, P.R., 2006. Comparison of conference abstracts and presentations with fulltext articles in the health technology assessments of rapidly evolving technologies. Health Technol. Assess. 10, iii–iv, ix–145. Fraile, L., Grau-Roma, L., Sarasola, P., Sinovas, N., Nofrarias, M., LopezJimenez, R., Lopez-Soria, S., Sibila, M., Segales, J., 2012. Inactivated PCV2 one shot vaccine applied in 3-week-old piglets: improvement of production parameters and interaction with maternally derived immunity. Vaccine 30, 1986–1992. Higgins, J.P.T., Green, S. (Eds.), 2011. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration (updated March 2011). Hopewell, S., McDonald, S., Clarke, M., Egger, M., 2007. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev, MR000010. Jansen, J.P., Fleurence, R., Devine, B., Itzler, R., Barrett, A., Hawkins, N., Lee, K., Boersma, C., Annemans, L., Cappelleri, J.C., 2011. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1. Value Health 14, 417–428. Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gotzsche, P.C., Ioannidis, J.P., Clarke, M., Devereaux, P.J., Kleijnen, J., Moher, D., 2009. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J. Clin. Epidemiol. 62, e1–e34.
95
Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., 2009. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J. Clin. Epidemiol. 62, 1006–1012. Morrison, D.E., Henkel, R.E., 2006. The Significance Test Controversy: A Reader. AldineTransaction, New Brunswick, NJ. Shea, B.J., Bouter, L.M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V.K., Grimshaw, J.M., 2007a. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2, e1350. Shea, B.J., Grimshaw, J.M., Wells, G.A., Boers, M., Andersson, N., Hamel, C., Porter, A.C., Tugwell, P., Moher, D., Bouter, L.M., 2007b. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med. Res. Methodol. 7, 10. Shea, B.J., Hamel, C., Wells, G.A., Bouter, L.M., Kristjansson, E., Grimshaw, J., Henry, D.A., Boers, M., 2009. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J. Clin. Epidemiol. 62, 1013–1020.
N. da Silva A. Carriquiry Department of Statistics, Iowa State University College of Liberal Arts and Sciences, Ames, IA 50011, United States K. O’Neill Department of Veterinary Diagnostics and Production Animal Medicine, Iowa State University College of Veterinary Medicine, 1600 South 16th Street, Ames, IA 50011, United States a
T. Opriessnig a,b Department of Veterinary Diagnostics and Production Animal Medicine, Iowa State University College of Veterinary Medicine, 1600 South 16th Street, Ames, IA 50011, United States b The Roslin Institute and The Royal (Dick) School of Veterinary Studies, University of Edinburgh, Easter Bush, Midlothian EH25 9RG, Scotland, UK A.M. O’Connor ∗ Department of Veterinary Diagnostics and Production Animal Medicine, Iowa State University College of Veterinary Medicine, 1600 South 16th Street, Ames, IA 50011, United States ∗ Corresponding
author at: Veterinary Medical Research Institute, Bld 4, College of Veterinary Medicine, Iowa State University, Ames, IA 50010, United States. Tel.: +1 515 294 5012; fax: +1 515 294 1072; mobile: +1 515 520 2376. E-mail address: [email protected] (A.M. O’Connor)
Contents lists available at ScienceDirect
Preventive Veterinary Medicine journal homepage: www.elsevier.com/locate/prevetmed
Reply to the Letter to the Editor Response to Allison et al. ‘Meta-analysis of porcine circovirus type 2 (PCV2) vaccines’ Thank you for the opportunity to respond to the letter submitted by Dr’s Allison, Taylor and Nitzel about the article “Mixed treatment comparison meta-analysis of porcine circovirus type 2 (PCV2) vaccines used in piglets. da Silva et al. (2014) Preventive Veterinary Medicine 117, 413–424” (da Silva et al., 2014). Among several issues raised, Dr’s Allison, Taylor and Nitzel provide a general opinion about the advantages of having more and better data for meta-analysis. We agree with this and ideally more higher-quality data would be better. This is a universal truth for any statistical analysis not just meta-analysis. Dr’s Allison, Taylor and Nitzel are perhaps suggesting that if faced with the same data they might not have conducted a meta-analysis. Clearly, we did not make such a decision and proceeded with the meta-analysis. Such decisions cannot be viewed as correct or incorrect, but are judgments researchers make and readers’ critique. Dr’s Allison, Taylor and Nitzel also have concerns about the sources of the studies used in the meta-analysis, the size of the studies and the approach to execution. We are pleased that the review is reported in a manner that enabled Dr’s Allison, Taylor and Nitzel to identify potential sources of bias. We strongly encourage assessment of bias when interpreting the primary research, and research synthesis is no exception. Several resources are available to encourage readers unfamiliar with critique of reviews (Shea et al., 2007a,b, 2009) and, more specifically, mixed treatment comparisons meta-analysis (Jansen et al., 2011). Research synthesis, like primary research, is a research endeavor and like primary research, there are potential sources of bias. Dr’s Allison, Taylor and Nitzel have pointed out what they consider to be biases. We are encouraged that the transparent reporting approach has allowed such critique and always encourage any end-user to assess the bias before deciding how to use the information provided. Such assessment of biases is sometimes not possible for reviews because critical information about the conduct of the review is missing such as studies included in the review, reasons for inclusion, exact estimates of outcomes and
DOI of the original article:http://dx.doi.org/10.1016/j.prevetmed. 2015.01.012. http://dx.doi.org/10.1016/j.prevetmed.2015.01.011 0167-5877/© 2015 Elsevier B.V. All rights reserved.
approaches to combining data. Some readers might feel the issues identified by Dr’s Allison, Taylor and Nitzel have minimal effect on the outcome; other readers might feel they are egregious; such debate is an integral part of the scientific process. Regardless we are pleased that our approach to reporting enabled such assessment, and encourage others conducting reviews to report in the same manner. We used the PRISMA statement when preparing our review report (Liberati et al., 2009; Moher et al., 2009). Dr’s Allison, Taylor and Nitzel express concern that use of conference proceedings is a potential source of publication bias “For some vaccines, but not others, sources included non-peer reviewed conference abstracts by company authors. Aside from concerns about the reliability of the experimental work, such material is highly subject to publication bias, with only positive trials presented.” Dr’s Allison, Taylor and Nitzel authors propose the direction of bias will be positive i.e., an over estimate of effect. However, we are unaware of documentation that suggests the direction of bias in veterinary vaccines studies and therefore prefer a transparent and inclusive approach to the review. In human health, conference proceedings are associated with less favorable outcomes (the direction opposite to that proposed by Dr’s Allison, Taylor and Nitzel) and therefore the inclusion of conference proceedings in human health clinical trial reviews is considered an approach to mitigating publication bias (Dundar et al., 2006; Hopewell et al., 2007). The Cochran Handbook of Systematic Reviews discusses the pro and cons of including gray literature and there is no consensus (Higgins and Green, 2011). In the area of swine health, the issue is unfortunately further complicated because there is evidence that peer-review publication is not common for vaccines, with less than 10% of conference proceedings subsequently published as peerreview articles. Therefore exclusion of these studies might simply have swapped one potential criticism for another (Brace et al., 2010). Our approach was to be transparent about the papers included and assess and report the risk of bias in all included studies which is consistent with recommended practices. Again, the decision to use data from conference proceedings is a judgment, and other readers might agree or disagree with the inclusion of such studies. Dr’s Allison, Taylor and Nitzel also express concern about our failure to use p values as decision making tools by saying that “Our concern with the “Highlights” is that they further enhance the impression that meaningful differences between the vaccines were shown when they were not.
Reply to the Letter to the Editor / Preventive Veterinary Medicine 119 (2015) 94–95
. . . without significant p-values for between-vaccine comparisons, no conclusion about between-vaccine differences is justified”. We would propose that estimation of effect size and reporting a precision of estimates is a more informative and appropriate approach to reporting and interpreting the data. We do not propose here to discuss the virtues of significance testing as entire books have been devoted to this topic (Morrison and Henkel, 2006). Our approach to reporting reflects the advice of the Cochran Handbook of Systematic Reviews of Interventions – Review authors are advised not to describe results as “not statistically significant” or “non-significant” (Higgins and Green, 2011). We respectably disagree that the highlights are misleading and believe that within the word limits these are a tempered and thoughtful summary of our opinion of the results. Finally after publication, we were asked to determine why the publication by Fraile et al. (2012) was not identified as relevant. Since publication, we have concluded that this manuscript was found by the search but was not passed from the screening team to the data extraction team, a false negative in the screening process. The authors did not report the number of animals in each group at each farm nor did they report if the measures of precision are standard errors of means (SEM) or standard deviation (SD). Thank you References Brace, S., Taylor, D., O’Connor, A.M., 2010. The quality of reporting and publication status of vaccines trials presented at veterinary conferences from 1988 to 2003. Vaccine 28, 5306–5314. da Silva, N., Carriquiry, A., O’Neill, K., Opriessnig, T., O’Connor, A.M., 2014. Mixed treatment comparison meta-analysis of porcine circovirus type 2 (PCV2) vaccines used in piglets. Prev. Vet. Med. 117, 413–424. Dundar, Y., Dodd, S., Dickson, R., Walley, T., Haycox, A., Williamson, P.R., 2006. Comparison of conference abstracts and presentations with fulltext articles in the health technology assessments of rapidly evolving technologies. Health Technol. Assess. 10, iii–iv, ix–145. Fraile, L., Grau-Roma, L., Sarasola, P., Sinovas, N., Nofrarias, M., LopezJimenez, R., Lopez-Soria, S., Sibila, M., Segales, J., 2012. Inactivated PCV2 one shot vaccine applied in 3-week-old piglets: improvement of production parameters and interaction with maternally derived immunity. Vaccine 30, 1986–1992. Higgins, J.P.T., Green, S. (Eds.), 2011. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration (updated March 2011). Hopewell, S., McDonald, S., Clarke, M., Egger, M., 2007. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev, MR000010. Jansen, J.P., Fleurence, R., Devine, B., Itzler, R., Barrett, A., Hawkins, N., Lee, K., Boersma, C., Annemans, L., Cappelleri, J.C., 2011. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1. Value Health 14, 417–428. Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gotzsche, P.C., Ioannidis, J.P., Clarke, M., Devereaux, P.J., Kleijnen, J., Moher, D., 2009. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J. Clin. Epidemiol. 62, e1–e34.
95
Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., 2009. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J. Clin. Epidemiol. 62, 1006–1012. Morrison, D.E., Henkel, R.E., 2006. The Significance Test Controversy: A Reader. AldineTransaction, New Brunswick, NJ. Shea, B.J., Bouter, L.M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V.K., Grimshaw, J.M., 2007a. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2, e1350. Shea, B.J., Grimshaw, J.M., Wells, G.A., Boers, M., Andersson, N., Hamel, C., Porter, A.C., Tugwell, P., Moher, D., Bouter, L.M., 2007b. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med. Res. Methodol. 7, 10. Shea, B.J., Hamel, C., Wells, G.A., Bouter, L.M., Kristjansson, E., Grimshaw, J., Henry, D.A., Boers, M., 2009. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J. Clin. Epidemiol. 62, 1013–1020.
N. da Silva A. Carriquiry Department of Statistics, Iowa State University College of Liberal Arts and Sciences, Ames, IA 50011, United States K. O’Neill Department of Veterinary Diagnostics and Production Animal Medicine, Iowa State University College of Veterinary Medicine, 1600 South 16th Street, Ames, IA 50011, United States a
T. Opriessnig a,b Department of Veterinary Diagnostics and Production Animal Medicine, Iowa State University College of Veterinary Medicine, 1600 South 16th Street, Ames, IA 50011, United States b The Roslin Institute and The Royal (Dick) School of Veterinary Studies, University of Edinburgh, Easter Bush, Midlothian EH25 9RG, Scotland, UK A.M. O’Connor ∗ Department of Veterinary Diagnostics and Production Animal Medicine, Iowa State University College of Veterinary Medicine, 1600 South 16th Street, Ames, IA 50011, United States ∗ Corresponding
author at: Veterinary Medical Research Institute, Bld 4, College of Veterinary Medicine, Iowa State University, Ames, IA 50010, United States. Tel.: +1 515 294 5012; fax: +1 515 294 1072; mobile: +1 515 520 2376. E-mail address: [email protected] (A.M. O’Connor)