- Email: [email protected]
Lessons from comparing narrative synthesis and meta-analysis in a systematic review
Meeting Abstracts
Lessons from comparing narrative synthesis and meta-analysis in a systematic review G J Melendez-Torres, James Thomas, Michelle Richardson, Lambert Felix, Theo Lorenc, Sian Thomas, Mark Petticrew
Abstract Background Narrative synthesis approaches have been faulted for vote-counting (in which effectiveness is determined by tallying direction and statistical significance of study results) and lack of transparency. Yet narrative syntheses can also highlight trends and explanations for findings in a way that statistical meta-analysis cannot. Debate about the absence of meta-analysis in situations where it could have been done continues in the methodological literature. Using a systematic review of the effects of plain packaging of tobacco products, we compared narrative syntheses and new meta-analyses. We chose this review because its findings, which informed UK public health policy, have been misrepresented by the tobacco industry. Methods We revisited a published systematic review on plain packaging of tobacco products. We compared the findings from the narrative synthesis originally used in the review that accounted for study quality with the findings from a new multilevel meta-analysis that included all available effect sizes. We compared them in terms of their inclusiveness, approach to heterogeneity, and overall findings. Findings Although 21 studies (n=27 166) were included in the narrative synthesis, we included seven studies in our new meta-analysis (56 effect sizes, n=5365). The narrative synthesis found that the 21 studies were “highly consistent” in the decreased attractiveness of plain packaging compared with branded packaging, primarily on the basis of direction and significance of effects as reported in the 21 studies. The pooled effect size for the seven studies in the meta-analysis demonstrated that plain packs were less attractive (Cohen’s d=–0·59, 95% CI –0·71 to –0·47). Whereas the narrative synthesis highlighted key generalities across subgroups and study designs, the meta-analysis highlighted possible effect modification by tobacco brand.
Published Online November 13, 2015 University of Warwick, Warwick, UK (G J Melendez-Torres DPhil); Institute of Education, University College London, London, UK (J Thomas PhD, M Richardson DPhil); London School of Hygiene & Tropical Medicine, London, UK (L Felix MSc, M Petticrew PhD); University of York, York, UK (T Lorenc PhD, S Thomas BA) Correspondence to: Dr G J Melendez-Torres, Department of Social Policy and Intervention, Barnett House, 32 Wellington Square, Oxford OX1 2ER, UK gerardo.melendez-torres@spi. ox.ac.uk
Interpretation This study adds to existing debates by showing the contribution and implications of each synthesis method, especially with respect to opportunities for exploring heterogeneity. The findings have implications for the credibility of different approaches. Even when narrative synthesis is preferred, a meta-analysis and forest plot can help generate additional hypotheses, test and confirm hypotheses, and understand heterogeneity. In this particular case, use of multilevel meta-analysis helped to make clear aspects of statistical heterogeneity that narrative synthesis alone might not have detected. Funding Medical Research Council. Contributors GJM-T carried out the analysis and prepared the findings. JT and MP conceived the analysis and helped in preparing the findings. All authors assisted in interpreting the analysis and preparing findings for presentation. Declaration of interests JT was an author of the original systematic review that formed the basis for this methodological investigation. All other authors declare no competing interests.
www.thelancet.com
9
Lessons from comparing narrative synthesis and meta-analysis in a systematic review G J Melendez-Torres, James Thomas, Michelle Richardson, Lambert Felix, Theo Lorenc, Sian Thomas, Mark Petticrew
Abstract Background Narrative synthesis approaches have been faulted for vote-counting (in which effectiveness is determined by tallying direction and statistical significance of study results) and lack of transparency. Yet narrative syntheses can also highlight trends and explanations for findings in a way that statistical meta-analysis cannot. Debate about the absence of meta-analysis in situations where it could have been done continues in the methodological literature. Using a systematic review of the effects of plain packaging of tobacco products, we compared narrative syntheses and new meta-analyses. We chose this review because its findings, which informed UK public health policy, have been misrepresented by the tobacco industry. Methods We revisited a published systematic review on plain packaging of tobacco products. We compared the findings from the narrative synthesis originally used in the review that accounted for study quality with the findings from a new multilevel meta-analysis that included all available effect sizes. We compared them in terms of their inclusiveness, approach to heterogeneity, and overall findings. Findings Although 21 studies (n=27 166) were included in the narrative synthesis, we included seven studies in our new meta-analysis (56 effect sizes, n=5365). The narrative synthesis found that the 21 studies were “highly consistent” in the decreased attractiveness of plain packaging compared with branded packaging, primarily on the basis of direction and significance of effects as reported in the 21 studies. The pooled effect size for the seven studies in the meta-analysis demonstrated that plain packs were less attractive (Cohen’s d=–0·59, 95% CI –0·71 to –0·47). Whereas the narrative synthesis highlighted key generalities across subgroups and study designs, the meta-analysis highlighted possible effect modification by tobacco brand.
Published Online November 13, 2015 University of Warwick, Warwick, UK (G J Melendez-Torres DPhil); Institute of Education, University College London, London, UK (J Thomas PhD, M Richardson DPhil); London School of Hygiene & Tropical Medicine, London, UK (L Felix MSc, M Petticrew PhD); University of York, York, UK (T Lorenc PhD, S Thomas BA) Correspondence to: Dr G J Melendez-Torres, Department of Social Policy and Intervention, Barnett House, 32 Wellington Square, Oxford OX1 2ER, UK gerardo.melendez-torres@spi. ox.ac.uk
Interpretation This study adds to existing debates by showing the contribution and implications of each synthesis method, especially with respect to opportunities for exploring heterogeneity. The findings have implications for the credibility of different approaches. Even when narrative synthesis is preferred, a meta-analysis and forest plot can help generate additional hypotheses, test and confirm hypotheses, and understand heterogeneity. In this particular case, use of multilevel meta-analysis helped to make clear aspects of statistical heterogeneity that narrative synthesis alone might not have detected. Funding Medical Research Council. Contributors GJM-T carried out the analysis and prepared the findings. JT and MP conceived the analysis and helped in preparing the findings. All authors assisted in interpreting the analysis and preparing findings for presentation. Declaration of interests JT was an author of the original systematic review that formed the basis for this methodological investigation. All other authors declare no competing interests.
www.thelancet.com
9