- Email: [email protected]
How can we synthesize qualitative and quantitative evidence for healthcare policy-makers and managers?
ORIGINAL ARTICLE
How can we synthesize qualitative and quantitative evidence for healthcare policy-makers and managers? by Catherine Pope, Nicholas Mays, and Jennie Popay
Catherine Pope is a Reader in Health Services Research at the School of Nursing and Midwifery at the University of Southampton where she undertakes primary research on healthcare organization and innovation, and professional practice.
Nicholas Mays is a health policy advisor to the New Zealand Treasury where he is a doer, user, and commissioner of reviews and Professor of Health Policy at the London School of Hygiene and Tropical Medicine.
Jennie Popay, Professor of Sociology and Public Health, University of Lancaster, has played an active part in encouraging the health research communities to include qualitative research within the ambit of systematic reviews of the effectiveness of healthcare. She established the Cochrane Qualitative Research Methods Group and the Campbell Implementation Methods Group.
Abstract Interest in synthesizing the findings of qualitative and quantitative evidence is increasing in response to the complex questions being asked by healthcare managers and policy-makers. There is a wealth of evidence available from many sources – both formal research and non-research based (e.g., expert opinion, stakeholder, and user views). Synthesis offers the opportunity to integrate diverse forms of evidence into a whole. We categorize the current approaches to the synthesis of qualitative and quantitative evidence into four broad groups: narrative, qualitative, quantitative, and Bayesian. Many of the methods for synthesis are emergent; some have been used to integrate primary data; few have a long history of application to healthcare. In the healthcare context, synthesis methods are less well developed than methods such as systematic review. Nonetheless, synthesis has the potential to provide knowledge and decision support to healthcare policymakers and managers. Résumé L’intérêt à résumer les conclusions des preuves qualitatives et quantitatives augmente en réponse aux questions complexes qui sont posées par les gestionnaires de soins de santé et les preneurs de décisions en matière de politiques. Il y a une pléiade de preuves provenant de nombreuses sources – certaines émanant de la recherche officielle et d’autres qui ne sont pas fondées sur les principes de la recherche (par ex., les opinions d’experts, les points de vue des intervenants et des utilisateurs). Le résumé offre la possibilité d’intégrer diverses formes de preuve en un tout. Nous catégorisons les approches actuellement disponibles pour résumer les preuves qualitatives et quantitatives en quatre grands groupes : les approches narratives, qualitatives, quantitatives et bayesiennes. Un grand nombre de méthodes de résumé sont en émergence et certaines ont été utilisées pour intégrer les données primaires; il n’y en a que peu qui ont une longue histoire d’application en soins de santé. Dans le contexte des soins de santé, les méthodes de mise en rapport sont moins bien élaborées que les méthodes comme l’examen systématique. Néanmoins la mise en rapport offre la possibilité de fournir un appui à la connaissance et un soutien à la décision pour les décideurs en matière de politiques et les gestionnaires de soins de santé.
S
ynthesis. noun. Building up; putting together; making a whole out of parts; the combination of separate elements of thought into a whole; reasoning from principles to a conclusion (opposite to analysis). Chambers Dictionary, 1992
Why synthesize research evidence? “Research synthesis” is in vogue. Methods for bringing together qualitative and quantitative evidence are being developed and debated; for example, the Canadian Health Services Research Foundation recently held a series of Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
27
Pope, Mays, and Popay
workshops on the subject1 and in collaboration with the U.K. National Health Service Delivery and Organisation Research and Development Programme, has commissioned three reports on the state of the art of synthesis.2,3,4 The topic has been the subject of reviews5 and was the focus of the first paper to appear in the newly launched Healthcare Policy.6 Interest in synthesizing the findings of qualitative and quantitative research follows in the wake of an exponential growth in the amount of research evidence available. Alongside the rise of evidence-based and evidence-informed practice, it has become accepted that “reviews of research are a better basis for informing policy than a single study or expert opinion.” 7 Research synthesis is seen as a way of developing reviews to incorporate diverse sources of evidence. The development of research synthesis is also a response to the complexity of questions being asked in healthcare – notably, questions asked by policy-makers and managers trying to make decisions about interventions, service delivery, and organization and system change. These decision-makers need to know about the causes of a problem and about the range of interventions that are theoretically possible. They ask questions about what works for whom and in what circumstances, and whether interventions will be acceptable to potential recipients. In addition, they want to consider the cost-effectiveness of different policy options. In the past, many systematic reviews focused on outcomes and prioritized particular types of quantitative evidence, typically those derived from randomized controlled trials. One definition of systematic review is “a summary of the medical literature that uses explicit methods to perform a thorough literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies.”8 However, more recently, reviewers, aware of the wide-ranging policy questions outlined above, have sought to include diverse sources of evidence, including qualitative research, and sometimes a variety of non-research sources, within systematic reviews.9,10 The term synthesis describes a process that combines parts into a whole. While “review” is literally a chance to “see again” (by bringing together a body of evidence from different sources), “synthesis” ideally moves beyond this and integrates evidence to produce something new. There is some controversy about whether synthesis is feasible or desirable. Some argue that synthesis destroys the integrity of individual studies, or that it is impossible because of the gap between different research approaches – quantitative research is typically associated with a positivist paradigm or perspective, while qualitative research is linked to a quite different, broadly interpretivist world view. While it is important to be aware of these different paradigms, we see qualitative-quantitative synthesis as a logical part of reviewing, and an activity that can offer considerable benefit to policy and healthcare decision-making.
28
TABLE 1. Stages in a review or synthesis • Defining the purpose • Specifying the question(s) • Scoping and mapping • Searching • Selection – inclusion/exclusion; quality; quantity
Approaches to the synthesis of qualitative and quantitative evidence The process of conducting a systematic review of the literature is adequately described elsewhere and will not be rehearsed in detail here11 and the stages in conducting a research synthesis mirror this process (see Table 1). That said, it is important to distinguish between reviews or syntheses undertaken to provide knowledge-support, which summarize the evidence as general background information for decision-making, and decision-support,12 where the review or synthesis performs some or all of the analytical tasks central to the decision-taking process in a particular context. Many synthesis methods are new or emerging and several were not explicitly developed for combining different types of evidence. Nonetheless, there are some approaches to synthesis that appear to offer potential for integrating qualitative and quantitative research evidence, and we categorize these in four broad groups: narrative, qualitative, quantitative, or Bayesian. Narrative approaches Narrative synthesis seeks to move beyond a summary of the research literature to generate new insights or knowledge using systematic and transparent methods. These approaches include work that integrates the findings of qualitative research with quantitative reviews or meta-analyses, for example, Harden et al.’s combination of a meta-analysis of trial data with a thematic analysis of qualitative studies to infer barriers and facilitators to healthy eating.10 Another example of a narrative synthesis approach is described by Greenhalgh et al.13 This review of the large and diverse literature on the diffusion of innovations employed meta-narrative mapping to unravel the “unfolding storyline of research within traditions.” Exploratory searching of literature was used to define 13 largely independent but coherent bodies of theoretical knowledge and linked empirical research (e.g., within “rural sociology”). These were referred to as traditions. A chronology and narrative or story was built up for each of these, and the findings from each tradition related to one another by identifying common themes, factors, and explanations. This provided the basis for a conceptual model, mapping all the evidence and showing empirical gaps as well as salient theories.
Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
HOW CAN WE SYNTHESIZE EVIDENCE ?
Figure 1. A model of medicine-taking behaviour.. Source: Pound P, et al., 200516 (copyright permission granted).
Figure 2. Total purchasing pilots by level of achievement. Source: Goodwin N, et al., 199820 (copyright permission granted).
Qualitative approaches These approaches to synthesis attempt to convert all the evidence into qualitative form (typically textual, although qualitative data may also be visual and/or oral) and to provide a non-numerical account – usually a narrative text – of the integration of that evidence. Quantitative data are transformed into qualitative form, for example, by extracting concepts or key findings. These approaches use qualitative methods of analysis, such as thematic analysis. Synthesis can be conducted using meta-ethnography or the qualitative cross-case methods. Cross-case analysis usually uses some
form of chart or matrix14 to display summarized material from several studies to facilitate comparison of the findings. Using such displays, it is possible to group together key concepts and identify core elements to develop new concepts or explanations.15 Meta-ethnography involves induction and interpretation (i.e., re-analysis) of the findings of research. This is achieved by reciprocal translation – a process similar to constant comparison in primary qualitative research, which entails examining the concepts in each study and looking for similarities and differences – in essence saying that “one study is like another except for x.” This analysis continues
Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
29
Pope, Mays, and Popay
until all the concepts have been translated into one another by matching, combining or adapting them, and from this it may be possible to develop a line of argument (a new theory or development of existing theory) or a refutational synthesis that can explain opposing interpretations in the literature. In this way, Pound et al.16 identified seven main groups of papers about medicine-taking related to specific medicines or conditions, and used reciprocal translation to develop the novel concept of patient “resistance” to medicines. This theorized that patients deliberately modify and adapt their medication-taking based on complex understandings and beliefs. Thus, non-adherence with drug regimens does not simply reflect a passive failure to take medicines, but is the result of active decision-making by the patient (see Figure 1). So far meta-ethnography has mainly been used to synthesize qualitative research17,18 but a recent approach, based on meta-ethnography, described as an interpretative synthesis19 has incorporated qualitative and quantitative literature on access to healthcare. Quantitative approaches Quantitative approaches to the synthesis of qualitative and quantitative findings from multiple studies integrate the research findings using statistical analysis methods. This means that all data need to be in numerical form, and thus any qualitative findings must be converted into quantitative data. One way of doing this uses content analysis – a technique for categorizing the data into themes that can then be counted and converted into frequencies to identify dominant issues across a number of studies. Another approach is to use structured questions to extract “observations” from a study that can be converted into numerical form and statistically analyzed. One example of how this can be done in primary research comes from the national evaluation of general practitioner “total purchasing pilots” in the U.K. National Health Service (NHS).20 Here the researchers summarized qualitative interview and other data to derive a “score” for each of the 52 sites in the study, which reflected its ability to bring about service change elsewhere in the local NHS through its activities as a purchaser of healthcare. The scores were examined against a range of characteristics of each site in a quantitative analysis and the sites were assigned to one of five hierarchical groups to help explain the relative success of the different pilots (see Figure 2). This work showed that the organizational type of total purchasing and the size of the organization were correlated with level of achievement. The same approach could be used to synthesize findings from different studies as opposed to different study sites in a single study. Bayesian approaches These approaches apply the principles of Bayesian analysis to synthesis and are particularly relevant to undertaking syntheses designed for decision support. One benefit of these approaches is that they can incorporate non-research sources of information such as expert or public opinion in
30
addition to research evidence. In such approaches the data from multiple studies are converted into quantitative form and pooled for analysis and modeling. An example of a synthesis that used this approach is a study using qualitative and quantitative evidence to assess factors affecting uptake of immunization.21 In this study, an analysis of qualitative studies of immunization uptake was used to inform a prior distribution (a numerical ranking of factors affecting immunization uptake from the individual studies). These derived prior probabilities were combined with probabilistic data from quantitative studies and analyzed to identify and rank a wide range of factors linked to uptake of immunization. Other forms of Bayesian-influenced approaches to synthesis include Bayesian approaches to cost-effectiveness analysis (CEA), which allow the use of non-trial evidence to inform decisions about the likely cost-effectiveness of particular treatments when they are applied in a specific context.22 Comprehensive decision-modeling can take this a step further, often building on a CEA, to incorporate the major steps in the decision process, ending with a recommended “best” policy option. By adopting such an approach, a wider range of evidence (non-research as well as research-based) can be incorporated into the synthesis along with the explicit value judgments necessary to identify the best course of action for decision-makers.12 Discussion Many of the methods for research synthesis are evolving.23 Several were developed as methods for integrating primary data, usually from a single methodological approach, be it qualitative or quantitative. Some, like meta-ethnography, have only relatively recently been applied to healthcare. As we have shown, there is a range of terminology in use, some of which can be confusing – terms such as narrative review, literature review, systematic review, and narrative synthesis are sometimes used interchangeably, but can refer to quite distinct approaches. There is no single unifying framework for synthesizing qualitative and quantitative evidence for healthcare policy-makers and managers, and in this sense the “rules” of how to do synthesis are less well developed than, say, those for conducting a systematic review of evidence on the effectiveness of an intervention. Perhaps the key messages are that it is important to choose one’s methods in light of the aims or questions of the synthesis, and to try to be as explicit and transparent in describing these methods as possible. For knowledge support, there are a range of synthesis approaches that might be employed, and the choice of method may depend on the nature of the question and the form of the evidence (qualitative or quantitative or both, and whether non-research sources are to be included). For decision support syntheses, there are fewer available methods and the synthesis will have to be clearly tailored to the demands of the decision-making process. It seems likely that the kinds of research synthesis required in a healthcare policy and management context will, for the moment at least, need to use a combination of approaches
Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
HOW CAN WE SYNTHESIZE EVIDENCE ?
so that a range of different types of evidence can be considered simultaneously. Given that qualitative and quantitative research often address quite different questions, it may be that in many cases a synthesis across the qualitative-quantitative boundary is unnecessary. The methods suggested above might simply be used to provide complementary reviews of particular types of evidence. Whatever the route, the aim for researchers, policy-makers, and managers is implicit in the definition of synthesis, that is, to move us from policy problems and goals towards a conclusion in terms of policy options.
References 1. http://www.chsrf.ca 2. Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policymaking in the health field. Journal of Health Services Research & Policy 2005;10(Supp 1):6-20. 3. Pawson R, Greenhalgh T, Harvey G, Walshe K. Using realist methods to produce syntheses of evidence for use by managers and policy makers. Journal of Health Services Research & Policy 2005;10(Supp l):21-34. 4. Lavis J, Davies H, Oxman A, Denis J-L, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. Journal of Health Services Research & Policy 2005;10(Supp l):35-48. 5. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: A review of possible methods. Journal of Health Services Research & Policy 2005;10:45-53. 6. Lomas J. Using research to inform healthcare managers and policy makers questions: From summative to interpretive synthesis. Healthcare Policy 2005;1:55-71. 7. Sheldon T. Making evidence synthesis more useful for management and policy making. Journal of Health Services Research & Policy 2005;10(Supp l):1-5. 8. http://www.jr2.ox.ac.uk/bandolier/booth/diagnos/glossary.html 9. Garcia J, Bricker L, Henderson J, Martin M-A, Mugford M, Nielson J, et al. Women’s views of pregnancy ultrasound: A systematic review. Birth 2002;29:225-250. 10. Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, et al. Applying systematic review methods to studies of people’s views: An example from public health research. Journal of Epidemiology and Community Health 2004;58:794-800. 11. Chalmers I, Altman DG. Systematic reviews. London: BMJ Books;1995.
12. Dowie J. The Bayesian approach to decision making. In: Killoran A, Swann C, Kelly M, editors. Public health evidence: Changing the health of the public. Oxford: Oxford University Press;(in press). 13. Greenhalgh T, Robert G, Macfarlane F, Bate SP, Kyriakidou O. Diffusion of innovations in service organizations: Systematic literature review and recommendations for future research. Milbank Quarterly 2004;82:581-629. 14. Miles MB, Huberman AM. Qualitative data analysis: An expanded sourcebook. London: Sage;1994. 15. Yin R. Case study research, design and methods. Applied Social Research Methods Series (Vol. 5). Thousand Oaks, CA: Sage;1984. 16. Pound P, Britten N, Morgan M, Yardley L, Pope C, DakerWhite G, et al. Resisting medicines: A synthesis of qualitative studies of medicine taking. Social Science and Medicine 2005;61:133-155. 17. Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, et al. Evaluating meta-ethnography: A synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science and Medicine 2003;56:671-684. 18. Walter FM, Emery J, Braithwaite D, Marteau T. Lay understanding of familial risk of common chronic diseases: A systematic review and synthesis of qualitative research. Ann Fam Med 2004;2:583-594. 19. Dixon Woods M, Kirk D, Agarwal S, Annandale E, Arthur T, et al. Vulnerable groups and access to health care: A critical interpretative synthesis. A report for the National Co-ordinating Centre for NHS Service Delivery and Organisation R&D (NCCSDO);2005. Available from: www.sdo.lshtm.ac.uk/pdf/ access_ dixon-woods_finalcopyedited.pdf 20. Goodwin N, Mays N, McLeod H, Malbon G, Raftery J, on behalf of the Total Purchasing National Evaluation Team (TPNET). Evaluation of total purchasing pilots in England and Scotland and implications for primary care groups in England: Personal interviews and analysis of routine data. BMJ 1998;317:256-259. 21. Roberts KA, Dixon-Woods M, Fitzpatrick R, Abrams KR, Jones DR. Factors affecting the uptake of childhood immunisation: A Bayesian synthesis of qualitative and quantitative evidence. Lancet 2002;360:1596-1599. 22. Luce BR, Claxton K. Redefining the analytical approach to pharmacoeconomics. Health Economics 1999;8:187-189. 23. Popay J (Editor). Moving beyond effectiveness: Methodological issues in the synthesis of diverse sources of evidence. London: National Institute for Health and Clinical Excellence;2006.
Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
31
How can we synthesize qualitative and quantitative evidence for healthcare policy-makers and managers? by Catherine Pope, Nicholas Mays, and Jennie Popay
Catherine Pope is a Reader in Health Services Research at the School of Nursing and Midwifery at the University of Southampton where she undertakes primary research on healthcare organization and innovation, and professional practice.
Nicholas Mays is a health policy advisor to the New Zealand Treasury where he is a doer, user, and commissioner of reviews and Professor of Health Policy at the London School of Hygiene and Tropical Medicine.
Jennie Popay, Professor of Sociology and Public Health, University of Lancaster, has played an active part in encouraging the health research communities to include qualitative research within the ambit of systematic reviews of the effectiveness of healthcare. She established the Cochrane Qualitative Research Methods Group and the Campbell Implementation Methods Group.
Abstract Interest in synthesizing the findings of qualitative and quantitative evidence is increasing in response to the complex questions being asked by healthcare managers and policy-makers. There is a wealth of evidence available from many sources – both formal research and non-research based (e.g., expert opinion, stakeholder, and user views). Synthesis offers the opportunity to integrate diverse forms of evidence into a whole. We categorize the current approaches to the synthesis of qualitative and quantitative evidence into four broad groups: narrative, qualitative, quantitative, and Bayesian. Many of the methods for synthesis are emergent; some have been used to integrate primary data; few have a long history of application to healthcare. In the healthcare context, synthesis methods are less well developed than methods such as systematic review. Nonetheless, synthesis has the potential to provide knowledge and decision support to healthcare policymakers and managers. Résumé L’intérêt à résumer les conclusions des preuves qualitatives et quantitatives augmente en réponse aux questions complexes qui sont posées par les gestionnaires de soins de santé et les preneurs de décisions en matière de politiques. Il y a une pléiade de preuves provenant de nombreuses sources – certaines émanant de la recherche officielle et d’autres qui ne sont pas fondées sur les principes de la recherche (par ex., les opinions d’experts, les points de vue des intervenants et des utilisateurs). Le résumé offre la possibilité d’intégrer diverses formes de preuve en un tout. Nous catégorisons les approches actuellement disponibles pour résumer les preuves qualitatives et quantitatives en quatre grands groupes : les approches narratives, qualitatives, quantitatives et bayesiennes. Un grand nombre de méthodes de résumé sont en émergence et certaines ont été utilisées pour intégrer les données primaires; il n’y en a que peu qui ont une longue histoire d’application en soins de santé. Dans le contexte des soins de santé, les méthodes de mise en rapport sont moins bien élaborées que les méthodes comme l’examen systématique. Néanmoins la mise en rapport offre la possibilité de fournir un appui à la connaissance et un soutien à la décision pour les décideurs en matière de politiques et les gestionnaires de soins de santé.
S
ynthesis. noun. Building up; putting together; making a whole out of parts; the combination of separate elements of thought into a whole; reasoning from principles to a conclusion (opposite to analysis). Chambers Dictionary, 1992
Why synthesize research evidence? “Research synthesis” is in vogue. Methods for bringing together qualitative and quantitative evidence are being developed and debated; for example, the Canadian Health Services Research Foundation recently held a series of Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
27
Pope, Mays, and Popay
workshops on the subject1 and in collaboration with the U.K. National Health Service Delivery and Organisation Research and Development Programme, has commissioned three reports on the state of the art of synthesis.2,3,4 The topic has been the subject of reviews5 and was the focus of the first paper to appear in the newly launched Healthcare Policy.6 Interest in synthesizing the findings of qualitative and quantitative research follows in the wake of an exponential growth in the amount of research evidence available. Alongside the rise of evidence-based and evidence-informed practice, it has become accepted that “reviews of research are a better basis for informing policy than a single study or expert opinion.” 7 Research synthesis is seen as a way of developing reviews to incorporate diverse sources of evidence. The development of research synthesis is also a response to the complexity of questions being asked in healthcare – notably, questions asked by policy-makers and managers trying to make decisions about interventions, service delivery, and organization and system change. These decision-makers need to know about the causes of a problem and about the range of interventions that are theoretically possible. They ask questions about what works for whom and in what circumstances, and whether interventions will be acceptable to potential recipients. In addition, they want to consider the cost-effectiveness of different policy options. In the past, many systematic reviews focused on outcomes and prioritized particular types of quantitative evidence, typically those derived from randomized controlled trials. One definition of systematic review is “a summary of the medical literature that uses explicit methods to perform a thorough literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies.”8 However, more recently, reviewers, aware of the wide-ranging policy questions outlined above, have sought to include diverse sources of evidence, including qualitative research, and sometimes a variety of non-research sources, within systematic reviews.9,10 The term synthesis describes a process that combines parts into a whole. While “review” is literally a chance to “see again” (by bringing together a body of evidence from different sources), “synthesis” ideally moves beyond this and integrates evidence to produce something new. There is some controversy about whether synthesis is feasible or desirable. Some argue that synthesis destroys the integrity of individual studies, or that it is impossible because of the gap between different research approaches – quantitative research is typically associated with a positivist paradigm or perspective, while qualitative research is linked to a quite different, broadly interpretivist world view. While it is important to be aware of these different paradigms, we see qualitative-quantitative synthesis as a logical part of reviewing, and an activity that can offer considerable benefit to policy and healthcare decision-making.
28
TABLE 1. Stages in a review or synthesis • Defining the purpose • Specifying the question(s) • Scoping and mapping • Searching • Selection – inclusion/exclusion; quality; quantity
Approaches to the synthesis of qualitative and quantitative evidence The process of conducting a systematic review of the literature is adequately described elsewhere and will not be rehearsed in detail here11 and the stages in conducting a research synthesis mirror this process (see Table 1). That said, it is important to distinguish between reviews or syntheses undertaken to provide knowledge-support, which summarize the evidence as general background information for decision-making, and decision-support,12 where the review or synthesis performs some or all of the analytical tasks central to the decision-taking process in a particular context. Many synthesis methods are new or emerging and several were not explicitly developed for combining different types of evidence. Nonetheless, there are some approaches to synthesis that appear to offer potential for integrating qualitative and quantitative research evidence, and we categorize these in four broad groups: narrative, qualitative, quantitative, or Bayesian. Narrative approaches Narrative synthesis seeks to move beyond a summary of the research literature to generate new insights or knowledge using systematic and transparent methods. These approaches include work that integrates the findings of qualitative research with quantitative reviews or meta-analyses, for example, Harden et al.’s combination of a meta-analysis of trial data with a thematic analysis of qualitative studies to infer barriers and facilitators to healthy eating.10 Another example of a narrative synthesis approach is described by Greenhalgh et al.13 This review of the large and diverse literature on the diffusion of innovations employed meta-narrative mapping to unravel the “unfolding storyline of research within traditions.” Exploratory searching of literature was used to define 13 largely independent but coherent bodies of theoretical knowledge and linked empirical research (e.g., within “rural sociology”). These were referred to as traditions. A chronology and narrative or story was built up for each of these, and the findings from each tradition related to one another by identifying common themes, factors, and explanations. This provided the basis for a conceptual model, mapping all the evidence and showing empirical gaps as well as salient theories.
Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
HOW CAN WE SYNTHESIZE EVIDENCE ?
Figure 1. A model of medicine-taking behaviour.. Source: Pound P, et al., 200516 (copyright permission granted).
Figure 2. Total purchasing pilots by level of achievement. Source: Goodwin N, et al., 199820 (copyright permission granted).
Qualitative approaches These approaches to synthesis attempt to convert all the evidence into qualitative form (typically textual, although qualitative data may also be visual and/or oral) and to provide a non-numerical account – usually a narrative text – of the integration of that evidence. Quantitative data are transformed into qualitative form, for example, by extracting concepts or key findings. These approaches use qualitative methods of analysis, such as thematic analysis. Synthesis can be conducted using meta-ethnography or the qualitative cross-case methods. Cross-case analysis usually uses some
form of chart or matrix14 to display summarized material from several studies to facilitate comparison of the findings. Using such displays, it is possible to group together key concepts and identify core elements to develop new concepts or explanations.15 Meta-ethnography involves induction and interpretation (i.e., re-analysis) of the findings of research. This is achieved by reciprocal translation – a process similar to constant comparison in primary qualitative research, which entails examining the concepts in each study and looking for similarities and differences – in essence saying that “one study is like another except for x.” This analysis continues
Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
29
Pope, Mays, and Popay
until all the concepts have been translated into one another by matching, combining or adapting them, and from this it may be possible to develop a line of argument (a new theory or development of existing theory) or a refutational synthesis that can explain opposing interpretations in the literature. In this way, Pound et al.16 identified seven main groups of papers about medicine-taking related to specific medicines or conditions, and used reciprocal translation to develop the novel concept of patient “resistance” to medicines. This theorized that patients deliberately modify and adapt their medication-taking based on complex understandings and beliefs. Thus, non-adherence with drug regimens does not simply reflect a passive failure to take medicines, but is the result of active decision-making by the patient (see Figure 1). So far meta-ethnography has mainly been used to synthesize qualitative research17,18 but a recent approach, based on meta-ethnography, described as an interpretative synthesis19 has incorporated qualitative and quantitative literature on access to healthcare. Quantitative approaches Quantitative approaches to the synthesis of qualitative and quantitative findings from multiple studies integrate the research findings using statistical analysis methods. This means that all data need to be in numerical form, and thus any qualitative findings must be converted into quantitative data. One way of doing this uses content analysis – a technique for categorizing the data into themes that can then be counted and converted into frequencies to identify dominant issues across a number of studies. Another approach is to use structured questions to extract “observations” from a study that can be converted into numerical form and statistically analyzed. One example of how this can be done in primary research comes from the national evaluation of general practitioner “total purchasing pilots” in the U.K. National Health Service (NHS).20 Here the researchers summarized qualitative interview and other data to derive a “score” for each of the 52 sites in the study, which reflected its ability to bring about service change elsewhere in the local NHS through its activities as a purchaser of healthcare. The scores were examined against a range of characteristics of each site in a quantitative analysis and the sites were assigned to one of five hierarchical groups to help explain the relative success of the different pilots (see Figure 2). This work showed that the organizational type of total purchasing and the size of the organization were correlated with level of achievement. The same approach could be used to synthesize findings from different studies as opposed to different study sites in a single study. Bayesian approaches These approaches apply the principles of Bayesian analysis to synthesis and are particularly relevant to undertaking syntheses designed for decision support. One benefit of these approaches is that they can incorporate non-research sources of information such as expert or public opinion in
30
addition to research evidence. In such approaches the data from multiple studies are converted into quantitative form and pooled for analysis and modeling. An example of a synthesis that used this approach is a study using qualitative and quantitative evidence to assess factors affecting uptake of immunization.21 In this study, an analysis of qualitative studies of immunization uptake was used to inform a prior distribution (a numerical ranking of factors affecting immunization uptake from the individual studies). These derived prior probabilities were combined with probabilistic data from quantitative studies and analyzed to identify and rank a wide range of factors linked to uptake of immunization. Other forms of Bayesian-influenced approaches to synthesis include Bayesian approaches to cost-effectiveness analysis (CEA), which allow the use of non-trial evidence to inform decisions about the likely cost-effectiveness of particular treatments when they are applied in a specific context.22 Comprehensive decision-modeling can take this a step further, often building on a CEA, to incorporate the major steps in the decision process, ending with a recommended “best” policy option. By adopting such an approach, a wider range of evidence (non-research as well as research-based) can be incorporated into the synthesis along with the explicit value judgments necessary to identify the best course of action for decision-makers.12 Discussion Many of the methods for research synthesis are evolving.23 Several were developed as methods for integrating primary data, usually from a single methodological approach, be it qualitative or quantitative. Some, like meta-ethnography, have only relatively recently been applied to healthcare. As we have shown, there is a range of terminology in use, some of which can be confusing – terms such as narrative review, literature review, systematic review, and narrative synthesis are sometimes used interchangeably, but can refer to quite distinct approaches. There is no single unifying framework for synthesizing qualitative and quantitative evidence for healthcare policy-makers and managers, and in this sense the “rules” of how to do synthesis are less well developed than, say, those for conducting a systematic review of evidence on the effectiveness of an intervention. Perhaps the key messages are that it is important to choose one’s methods in light of the aims or questions of the synthesis, and to try to be as explicit and transparent in describing these methods as possible. For knowledge support, there are a range of synthesis approaches that might be employed, and the choice of method may depend on the nature of the question and the form of the evidence (qualitative or quantitative or both, and whether non-research sources are to be included). For decision support syntheses, there are fewer available methods and the synthesis will have to be clearly tailored to the demands of the decision-making process. It seems likely that the kinds of research synthesis required in a healthcare policy and management context will, for the moment at least, need to use a combination of approaches
Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
HOW CAN WE SYNTHESIZE EVIDENCE ?
so that a range of different types of evidence can be considered simultaneously. Given that qualitative and quantitative research often address quite different questions, it may be that in many cases a synthesis across the qualitative-quantitative boundary is unnecessary. The methods suggested above might simply be used to provide complementary reviews of particular types of evidence. Whatever the route, the aim for researchers, policy-makers, and managers is implicit in the definition of synthesis, that is, to move us from policy problems and goals towards a conclusion in terms of policy options.
References 1. http://www.chsrf.ca 2. Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policymaking in the health field. Journal of Health Services Research & Policy 2005;10(Supp 1):6-20. 3. Pawson R, Greenhalgh T, Harvey G, Walshe K. Using realist methods to produce syntheses of evidence for use by managers and policy makers. Journal of Health Services Research & Policy 2005;10(Supp l):21-34. 4. Lavis J, Davies H, Oxman A, Denis J-L, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. Journal of Health Services Research & Policy 2005;10(Supp l):35-48. 5. Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: A review of possible methods. Journal of Health Services Research & Policy 2005;10:45-53. 6. Lomas J. Using research to inform healthcare managers and policy makers questions: From summative to interpretive synthesis. Healthcare Policy 2005;1:55-71. 7. Sheldon T. Making evidence synthesis more useful for management and policy making. Journal of Health Services Research & Policy 2005;10(Supp l):1-5. 8. http://www.jr2.ox.ac.uk/bandolier/booth/diagnos/glossary.html 9. Garcia J, Bricker L, Henderson J, Martin M-A, Mugford M, Nielson J, et al. Women’s views of pregnancy ultrasound: A systematic review. Birth 2002;29:225-250. 10. Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, et al. Applying systematic review methods to studies of people’s views: An example from public health research. Journal of Epidemiology and Community Health 2004;58:794-800. 11. Chalmers I, Altman DG. Systematic reviews. London: BMJ Books;1995.
12. Dowie J. The Bayesian approach to decision making. In: Killoran A, Swann C, Kelly M, editors. Public health evidence: Changing the health of the public. Oxford: Oxford University Press;(in press). 13. Greenhalgh T, Robert G, Macfarlane F, Bate SP, Kyriakidou O. Diffusion of innovations in service organizations: Systematic literature review and recommendations for future research. Milbank Quarterly 2004;82:581-629. 14. Miles MB, Huberman AM. Qualitative data analysis: An expanded sourcebook. London: Sage;1994. 15. Yin R. Case study research, design and methods. Applied Social Research Methods Series (Vol. 5). Thousand Oaks, CA: Sage;1984. 16. Pound P, Britten N, Morgan M, Yardley L, Pope C, DakerWhite G, et al. Resisting medicines: A synthesis of qualitative studies of medicine taking. Social Science and Medicine 2005;61:133-155. 17. Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, et al. Evaluating meta-ethnography: A synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science and Medicine 2003;56:671-684. 18. Walter FM, Emery J, Braithwaite D, Marteau T. Lay understanding of familial risk of common chronic diseases: A systematic review and synthesis of qualitative research. Ann Fam Med 2004;2:583-594. 19. Dixon Woods M, Kirk D, Agarwal S, Annandale E, Arthur T, et al. Vulnerable groups and access to health care: A critical interpretative synthesis. A report for the National Co-ordinating Centre for NHS Service Delivery and Organisation R&D (NCCSDO);2005. Available from: www.sdo.lshtm.ac.uk/pdf/ access_ dixon-woods_finalcopyedited.pdf 20. Goodwin N, Mays N, McLeod H, Malbon G, Raftery J, on behalf of the Total Purchasing National Evaluation Team (TPNET). Evaluation of total purchasing pilots in England and Scotland and implications for primary care groups in England: Personal interviews and analysis of routine data. BMJ 1998;317:256-259. 21. Roberts KA, Dixon-Woods M, Fitzpatrick R, Abrams KR, Jones DR. Factors affecting the uptake of childhood immunisation: A Bayesian synthesis of qualitative and quantitative evidence. Lancet 2002;360:1596-1599. 22. Luce BR, Claxton K. Redefining the analytical approach to pharmacoeconomics. Health Economics 1999;8:187-189. 23. Popay J (Editor). Moving beyond effectiveness: Methodological issues in the synthesis of diverse sources of evidence. London: National Institute for Health and Clinical Excellence;2006.
Healthcare Management FORUM Gestion des soins de santé – Spring/Printemps 2006
31