Using systematic review methods to evaluate environmental public policy: methodological challenges and potential usefulness

Environmental Science and Policy 105 (2020) 47–55

Contents lists available at ScienceDirect

Environmental Science and Policy journal homepage: www.elsevier.com/locate/envsci

Using systematic review methods to evaluate environmental public policy: methodological challenges and potential usefulness

T

Matilda Miljand Department of Political Science, Umeå University, 901 87 Umeå, Sweden

A R T I C LE I N FO

A B S T R A C T

Keywords: Public policy evaluation Environmental public policy Systematic review methodology Synthesis methods Context

This article provides an overview of how systematic review (SR) methods have been used to evaluate public policy. It argues that these methods can be applied to the evaluation of environmental public policy, but that certain challenges need to be addressed in order to fulfil the SR methods potential. The article reflects upon two methodological challenges confronting systematic reviewers: how data from the articles should be synthesised; and how to take societal contexts into account. Analysing how these challenges have been addressed in practice contributes to the theoretical discussion about the usefulness of different synthesis methods, and the role of context. Three lessons are drawn as to how systematic review methods can become useful when applied to the evaluation of environmental public policy, namely: (1) to anticipate the heterogeneity in the literature from the beginning – in terms of both research design and operationalisation of key concepts; (2) to consider the purpose of the review when deciding whether to take a single- or multi-context approach; and (3) to be methodologically innovative when applying the systematic review methods to complex policies.

1. Introduction Both scientists and policy-makers have recognised the need to evaluate environmental public policy. This has led to a demand for methods that can incorporate the complexity of environmental policies (Gysen et al., 2006; Mickwitz, 2003; Weber et al., 2014). A group of methods called systematic reviews (SR) have become commonly used when assessing the effects of interventions primarily in healthcare and social work, but also increasingly to assess the impact of interventions in the environment. It is therefore interesting to investigate the potential for using of these methods to evaluate environmental public policy. SR is an umbrella term for a set of review methods that involves “a review of the research literature using systematic and explicit accountable methods” (Gough et al., 2012 p. 5). Common to all SRs is the use of explicit methods for selecting and analysing empirical data. Traditional SR is guided by the idea of repeatability, objectivity and comprehensiveness - including the search for and selection of literature. SR methods are often contrasted with ‘literature reviews’. A literature review does not prescribe a specific method as it is a generic term for a review that provides an examination of recent or current literature. This can “cover a wide range of subjects at various levels of completeness and comprehensiveness” (Grant and Booth, 2009 p. 94). All literature reviews have certain commonalities with SRs. They may include

comprehensive searching or a quality assessment of the studies included. It is common that the literature is synthesised narratively and analysed conceptually or thematically (Grant and Booth, 2009), but literature reviews may also synthesise the results using a meta-analysis. Therefore, it can be difficult to draw a sharp line for when a literature review is a SR. SR methods have become increasingly used in environmental science to compile scientific evidence in order to inform environmental decision-making. In the early 2000s a framework for attaining evidence-based conservation was introduced (Pullin and Knight, 2003; Pullin et al., 2004; Pullin and Knight, 2005), joining the so-called evidence movement that had started in medicine a decade earlier. This has been followed by methodological development of how SR methods can be used to synthesise environmental science research (Bilotta et al., 2014), including discussion on how the knowledge gained through SR can impact policy decisions (Wyborn et al., 2018; Pullin et al., 2009). This article shifts the focus from synthesising environmental science research to the potential of synthesising research on environmental public policy. SR methods are relatively rare in two areas where public policy evaluation is common: political science (Dacombe, 2017) and public administration research (Overman, 2016). However, there is reason to believe that the SR methodology can contribute to these areas by making literature reviews more transparent and by improving the explanatory potential through the use of statistical methods such as

E-mail address: [email protected]. https://doi.org/10.1016/j.envsci.2019.12.008 Received 11 April 2019; Received in revised form 7 October 2019; Accepted 16 December 2019 1462-9011/ © 2019 Elsevier Ltd. All rights reserved.

Environmental Science and Policy 105 (2020) 47–55

M. Miljand

et al., 2013), and thematic analysis (Thomas and Harden, 2008). Realist reviews can be considered to have both an aggregative and a configurative aim. It is a method for evaluating and developing comprehensive programme theories (Pawson et al., 2004, 2005) which unpack the mechanisms of how and why complex interventions thrive or fail in particular settings. Regardless of synthesis method all reviews should contain a narrative summary of the evidence base. This entails a description of the literature and should not be confused with qualitative synthesis. In a narrative summary the main study characteristics and results are summarised, often in table format. These summaries “vary from the simple recounting and description of findings through to more interpretive and explicitly reflexive accounts that include commentary and higher levels of abstraction” (Dixon-Woods et al., 2005 p. 47). In this overview systematic map methodology is included as a form of narrative summary. Systematic mapping is identical to traditional SR with the exception that is does not include any synthesis (and sometimes no quality assessment).

meta-analysis. 1.1. The aim of this overview In this study I focus on methodological problems that might arise when applying SR methods to public policy. Two methodological issues are discussed in particular: firstly, which synthesis methods are suitable to use for what purposes; and secondly, the question of how to approach context. The latter, how to account for difference in context, is crucial when investigating the applicability of the SR methodology to public policy evaluation as public policy evaluators face the considerable challenge of the social contexts within which the policy will be implemented (Hanberger, 2001). A way to move beyond the theoretical discussion of the usefulness of SR methods is to learn from what is currently done. At present, there lacks a literature focusing on how SR methods have been used in public policy evaluations. The aim of this study is to fill that void. More specifically, three research questions are posed: (1) How are different synthesis methods used in SRs of public policy?; (2) How do the reviewers relate to and account for context in SRs of public policy?; and 3) What lessons can be drawn on how to meet the aforementioned methodological challenges when evaluating environmental public policy? The key concepts in this article must firstly be clarified. The term ‘overview’ refers to this particular article. ‘Systematic review’ (SR) refers to the method as well as the individual SRs that constitute the empirical material for this overview. Finally, SRs are reviews of literature and therefore based on many different studies. The article proceeds as follows, it begins by presenting different approaches to SR, including the synthesis methods and approaches to context. I then describe the method used in this overview, followed by a presentation and discussion of the results. Finally, conclusions are drawn on the applicability of SR methods when evaluating environmental public policy.

2.2. Single- versus multi-context reviews The importance of context to evaluation practice has concerned many evaluation theorists (c.f. Krogstrup, 2003; Pawson, 2006). Here, context refers to the setting within which the policy, programme or intervention is situated, e.g. demographic characteristics of the setting or the people in it, economic or institutional features, and/or political dynamics. In 2011, Hannes and Harden highlighted this issue in relation to (qualitative) SRs by discussing single-context and multi-context reviews. Booth et al. (2019 p. 3) find that "the implications of conducting single- or multi-contexts are poorly researched and, consequently poorly understood". However, they note that review methodologists have begun to discuss the merits of these different reviews approaches. Although Booth et al. (2019) discuss the role of context in nursing, with a focus on qualitative evidence synthesis, their reasoning has a much broader relevance. They argue that single-context reviews can help us understand how changes within a specific context affect an intervention, while multi-context reviews help us understand how "appropriateness, meaningfulness and perceived effectiveness of an intervention differ across contexts" (Booth et al., 2019 p. 2). They identify strengths and limitations with both approaches. Strengths of singlecontext reviews include facilitating contextual sensitivity and reveal disconfirming cases that facilitate a more nuanced understanding. However, limitations include: the need for local and regional expertise; the risk that the results may prove less generalisable; and that the review may risk missing the "big picture". When it comes to multi-context reviews, strengths include the transferability of results across geopolitical barriers and the potential for revealing contextual variation in how the phenomenon of interest is conceptualised and/or operationalised. Limitations include a review team that may lack local and regional contextual knowledge, and the blurring of important distinctions in definitions or interpretations across studies and contexts (Booth et al., 2019). Booth et al. (2019) argue that the decision whether to carry out a single- or multi-context review should be informed by the research question, the audience and purpose of the review, and the type of data. Applying this logic to the question of public policy suggests that if the aim is to understand how public policy functions within a specific context, single-context review is the most appropriate. However, if the aim is to understand how a given policy works across different contexts, and the exploration of such variation, a multi-context review is preferable.

2. Approaches to systematic reviews 2.1. Different systematic review methods The last decades have seen a proliferation of SR methods and several articles provide a description of different methods (e.g. Barnett-Page and Thomas, 2009; Dixon-Woods et al., 2005; Gough et al., 2012; Grant and Booth, 2009; Hannes and Lockwood, 2011; Macura et al., 2019). Some review methods focus on aggregating the results of primary studies. An example of such a method is traditional SR (Collaboration for Environmental Evidence, 2013; Gough et al., 2012) in which statistical methods are used to aggregate the results from primary research. Data is pooled and re-analysed using meta-analysis. Empirical data can also be grouped to answer a question (Gough et al., 2012) and this is done in a meta-aggregation review (Hannes and Lockwood, 2011) whereby results from primary research are combined to give a description of the phenomenon being studied. Another aggregating ambition is found in content analysis (Dixon-Woods et al., 2005; Mays et al., 2001) where studies are systematically categorised, after which the frequencies for these categories are calculated. Other methods focus predominantly on configuring or interpreting the findings from primary studies. These reviews tend to be associated with questions that generate new theory (Gough et al., 2012). These methods focus on the use of different concepts, metaphors or themes across different studies. The reviewer compares and contrasts the different concepts, metaphors or themes and creates new concepts or more analytical themes. This then provides new interpretations, explanations or hypotheses. Examples of methods with a configurative aim are metaethnography (Campbell et al., 2012; Noblit and Hare, 1988), critical interpretive synthesis (Dixon-Woods et al., 2006b), grounded theory (Dixon-Woods et al., 2005; Eaves, 2001; Kearney, 2001; Wolfswinkel

3. Method applied in the overview In this overview, SRs that have evaluated public policy are compiled 48

Environmental Science and Policy 105 (2020) 47–55

M. Miljand

The included SRs make up the empirical data of this overview. The SRs that were found to be relevant were imported into NVivo 11, a qualitative data analysis software designed to organise and analyse qualitative data. In NVivo 11, the SRs were categorised according to how they dealt with the two methodological challenges of synthesis of data and accounting for context. Each SR’s aim and synthesis method and contextual limitations was identified.

and analysed. Before turning to the results of this analysis, the method for the overview is presented. An overview refers to a summary of the literature and describes its characteristics (Grant and Booth, 2009). In order to select relevant studies, the following definitions were applied. While there is no generally accepted definition of SRs, there is agreement that certain criteria need to be fulfilled in order to classify as SR. Namely, clear description of method, systematic search (though not necessarily comprehensive), systematic screening process (i.e. predefined inclusion criteria), and explicit form of synthesis. Public policy refers to the “efforts by which public authorities wield their power in attempting to affect society in terms of values and beliefs, action and organisation” (Mickwitz, 2006 p. 9). Public policy is thus here defined as a course of action that involves policy instruments such as regulation, economic incentives and/or information introduced by a government or legislature (international, national or local) but not by non-government organisations or private companies (e.g. individual hospitals or schools). The way a policy is organised or governed was also included in the selection of studies. Consequently, searches were conducted for SRs that talked about the governance of a particular policy area. Governance is taken here as the organisation of how decisions are made and the sharing of power across societal levels. It is "the way in which rights and responsibilities, initiatives, checks and balances are divided between public and private realm, between central and local as well as national and international” (Christiansen, 1996 p. 17). Of particular interest are reviews that examine how public governance is designed and functioning, rather than, for example, how individual hospitals are governed or how individual hospitals, schools, or national parks are set up and/or managed. During the work with this overview it became clear that making a sharp distinction of what should be categorised as a public policy or governance review is not straightforward. Hence, the included reviews should not be considered as an exhaustive list. Rather they represent a reasonable account of the available literature with the purpose to learn how different methods have been applied and what methodological problems that the reviewers have encountered and discussed. Including different types of policy reviews was more important than comprehensiveness in this case. Searches to identify relevant SRs were made in the libraries of organisations publishing SR, such as Cochrane Collaboration Library, Campbell Collaboration Library/3ie, and Environmental Evidence Journal. The Web of Science database was also used to search for academic literature. Table 1 presents the databases and the search terms used (see Appendix A for full details of the searches). Campbell Library and Environmental Evidence Journals library were limited enough to allow for all published reviews to be screened. In Cochrane Library and Web of Science searches strings were used. All searches were carried out in English. The screening process for selecting the relevant SRs was done stepwise. The SRs were first screened by title, and if deemed potentially relevant, the abstracts were subsequently screened. Full text SRs articles were obtained for all potentially relevant abstracts. When it was unclear whether a particular SR was relevant, it was included for further assessment. Only complete, published SRs were included, which means that review protocols were excluded.

4. Findings 4.1. The use of SR methods to evaluate public policy This section presents the results of the overview. The literature searches resulted in 1 919 articles, out of which 26 SRs of public policy were deemed relevant to be included in the analysis presented below (for a detailed description see Appendix B). Table 2 presents the screening process according to the searched databases. The first result highlights the development period of SRs. The earliest relevant SR was published in 2005, and a large majority (n = 21) have been published after 2014 indicating that it is a fairly recent development using SR methods to evaluate public policy. The identified SRs cover different research areas with a majority being in health (n = 10), environment/natural resource management (n = 9), followed by two in social policy, and one each in international development, transportation, agricultural policy, education, and crime/justice. A detailed description of these reviews is given in Appendix A. 4.2. The use of different synthesis methods The final step in a SR is to synthesize empirical data and findings from the identified studies. Four of the studies applied quantitative synthesis methods and two used a qualitative synthesis method. The majority (20 out of 26) of the identified SRs, have only narratively summarised the included studies. Cirera et al. (2014) discuss the use of different outcome variables make it hard to pool the data into one analysis, they show that assessing how effects vary among outcome measurement can provide insights as to why different studies come to different conclusions. The review by Cirera et al. (2014) is therefore an illustrative example of both the challenges with SR in general and meta-analysis in particular. In six reviews the authors set out to do a quantitative synthesis but then discovered this was not possible. The included studies were either too heterogeneous or lacked statistical detail to be able to be synthesised in a meta-analysis (Dugan et al., 2013; Hall, 2017; McPhedran, 2016; Samii et al., 2014). In the case of Peñaloza et al. (2011) only one study was included in the final SR precluding synthesis, and instead the data was presented in a narrative format; and Macura et al. (2015) concluded that mapping the existing evidence, i.e. presenting the evidence and describing the included studies was “a more appropriate approach for the topic area, which appeared too broad and divergent for a single systematic review exercise”. Fourteen reviews (Aitken et al., 2015; Auld et al., 2014; Bambra et al., 2005, 2014; Brisbois and de Loe, 2016; Howes et al., 2017; Hyde and Shortell, 2012; Hyseni et al., 2017; Lwasa, 2015; Meng et al., 2011; Meurk et al., 2016; O’Reilly et al., 2017; Rode et al., 2015; Sud et al., 2015) had initially decided to do narrative summaries. In these, the existing literature relating to the topic of choice is described. Some only provide a short overview, with limited analysis of the evidence. Others attempt to make sense of the literature they have identified. Three examples of how to carry out a narrative summary are given in Table 3. The overview illustrates that narrative summary (n = 20) is most common when it comes to SRs of public policy. It is reasonable to argue that synthesising diverse data on public policy renders quantitative synthesis particularly complicated, and thus narrative summary often becomes more appropriate.

Table 1 Search terms. Databases

Search terms 1/ Search terms 2

Campbell Collaboration Library/ 3ie Cochrane Collaboration Library Environmental Evidence Journal Web of Science

(Screened all) Policy/ Governance (Screened all) Systematic review AND Policy/ Systematic review AND Governance

49

Environmental Science and Policy 105 (2020) 47–55

M. Miljand

Table 2 Search results and screening process by database. Database

Hits

Excluded at title

Excluded at abstract

Excluded at full text

Included in final analysis

Web of Science* Cochrane Campbell Collaboration Library/3ie Environmental Evidence Journal Total

1445 318 128 28 1919

1111 278

231 32 123 26 412

81 7 3 1 92

22 1 2 1 26

1389

* The searches were limited to potentially relevant subject categories. For detailed information see Appendix A.

4.3. Single- versus multi-context reviews

Table 4 Different limitations of contextsa.

How and to what extent context is taken into account in a SR depends on the choice of synthesis method and the available data. The included reviews have taken both a single- and multi-context approach, see Table 4. Several authors raise the question of how evidence from different contexts can be synthesised appropriately. Several of the SRs originally intended to include contextual factors into their analysis in order to assess their impact on outcomes, with the aim to gain a deeper understanding of why outcomes vary across studies. However, a common conclusion was that lack of data prevented such analyses. Lawry et al. (2014) looked at contextual factors that could potentially moderate the effects of tenure recognition. They found that the “evidence base is too thin to say how productivity and investment effects are moderated by our other contextual factors of interest” (Lawry et al., 2014 p. 9), but concluded that there are regional differences and that the gains in productivity are significantly greater outside Africa and in wealthier settings. Another example is Samii et al. (2014). They collected data on the study characteristics, findings, and moderators of all included studies. However, they were unable to carry out any meta-regressions to test the effects of these moderators and mediators due to the wide range of outcome variables, instead relying

CONTEXT

NUMBER OF REVIEWS

One country Limitations connected to rivers/deltas Limitation to high, middle and/or low income countries No limitations

6 2 7 11

a

These are the limits imposed by the reviewers before searching and screening the literature. It does not mean that all countries are included in the final review. The literature might not contain information on all the countries that are eligible to be included.

on “qualitative information relevant for the included studies to comment on, rather than test, the moderator and mediator hypotheses” (Samii et al., 2014 p. 31). These examples illustrate the need for extensive data to quantitatively account for contextual differences and suggest that such information is often missing. Choosing a single-context approach, and limiting the focus of the study to one political context, was one way that the SRs overcame the challenges of comparing different public policy contexts. Six SRs

Table 3 Summary of variation of synthesis methods in the examined reviews. SYNTHESIS METHOD

NO. OF REVIEWS

EXAMPLES OF METHOD USED

Quantitative synthesis Meta-analysis

3

Meta-regression

1

Meta-analysis is a statistical analysis that combines the results of multiple scientific studies and can be performed when multiple scientific studies address the same question. Filges et al. (2015) assess the impact of exhaustion of unemployment benefits. They performed the meta-analyses on the included studies using the log HR, where HR measures the proportional change in hazard rates between unemployed persons approaching exhaustion and unemployed persons not approaching exhaustion. Lawry et al. (2014) assessed the impacts of interventions to strengthen land property rights on agricultural and livelihood. They carried out a meta-regression on a sets of comparably measured outcomes: productivity of land, consumption/income, probability of formal borrowing, amount of formal credit obtained, probability of perceiving land may be expropriated and investment in long-term production.

Qualitative synthesis Critical interpretive synthesis (CIS)

1

Grounded theory

1

Reviews without synthesis Narrative summary Systematic map

20

Presenting themes in the literature

Aggregative meta-summary

The discussion in Perkins et al. (2010) is inspired by CIS. It involves thematically exploring the theories developed in the various studies, in addition to focusing on the empirical results that are described. Most of the studies examined process, rather than outcome, issues and this is reflected in the evidence synthesis in which there are six themes relating to process issues and only two relating to health outcomes. Pittman and Armitage (2016) used grounded theory when analysing the literature on the topic of governance across the land sea interface. The analysis was carried out in three steps: iterative rounds of open coding to capture all possible themes found in the data, axial coding to identify patterns and relationships within and between themes and finally selective coding to refine theoretical constructs through compilation of evidence.

Macura et al. (2015) mapped all studies found relevant in the literature screening using a predefined coding scheme based on the research question. Information was extracted on various aspects of study setting and design, type of methodology used, type of governance, description of outcomes and comparators. A searchable systematic map database was created. Lwasa (2015) synthesises literature on adaptation policy and practice in river deltas located in Africa and South Asia. The included studies were categorised and information about publication type, date, country and authors extracted and presented. This was used to show the focus of the included studies, as well as whether the studies agreed on policy and practice. Lawry et al. (2014) used the qualitative ‘metasummary’ methodology of Sandelowski et al. (2007). Metasummaries involve a process of extracting findings, grouping them into themes and then establishing the frequency and intensity of findings corresponding to the themes. The results are put into a matrix to convey the themes and their frequency and intensity.

50

Environmental Science and Policy 105 (2020) 47–55

M. Miljand

methodological challenges of data synthesis and approaches to context have been met. The first is to anticipate the heterogeneity in the literature in terms of initial research design and operationalisation of key concepts. The concepts that are investigated in public policy research are generally not straightforward in how they can be operationalised or measured. A public policy in one country differs in terms of design and implementation from a similar public policy in another country. This knowledge can be used to make initial decisions about the scope of the review and the appropriate synthesis method. It should also inform decisions on what literature is relevant to include in the review. The second lesson regards the need to think critically about whether a single- or multi-context review is the most appropriate approach. A guiding principle should be that if the purpose is to evaluate the effect of a policy within a particular context then a single-context review is preferable, while if the purpose is to better understand how the effect of a policy varies across context then a multi-context approach is preferred. This principle points to the importance of reflecting on one's purpose with a review in relation to context situation. If a multi-context approach is taken, it is important to think about how to account for contextual differences when analysing the data. Data availability may affect the possibility of analysing cross context difference. This overview has shown considerable challenges in this respect. A third, and maybe most important, lesson is to be methodologically innovative when applying the SR method to complex policies, such as environmental. This is also emphasised by Forss et al. (2011) in their examination of how to evaluate complex interventions or policies. They argue both for methodological innovation when trying to determine causality in a complex environment and for assessing the merit of a policy by using many perspectives in a flexible manner. This overview supports such an approach, and displays the variety of available synthesis methods. The methodological diversity among the SR methods are a strength to be used if SRs are to be used to evaluate environmental public policy. Looking ahead into how SRs could be used to inform decisions in public policy, it may be worth considering what types of questions are pertinent for decision makers and at the same time suitable for SR methods. As mentioned above, an important challenge in using SR to answer policy questions is the heterogeneous data that makes synthesis difficult. Additionally, it may be of interest to decision makers to gain insight into how similar policies have been implemented in different countries and what different challenges were encountered. These lines of engagement can help them think more creatively about the possible options and solutions and to answer such, heterogeneity can be a strength. Different studies can contribute with different aspects that have proven to influence the implementation and outcome of a policy. This can provide valuable policy making insights even if it means that the effects of a specific policy cannot be assessed. Finally, the value of using SR methods in order to understand different environmental policies should not be underestimated for researchers themselves. SR can contribute with a deeper understanding of the research field, which methods and perspectives are applied and which are missing. SR can then be used to develop more comprehensive theories. There is also a value in developing new synthesis methods, especially when it comes to reviews of public policy. It therefore holds to make use of common methods in political science by drawing inspiration from case study methods and comparative methods when developing new synthesis methods.

focused only on one specific country; two on the US, two on the UK, and two on Australia. The focus on a single country was motivated by Dugan et al. (2013 p. 350) due to "the importance of social and regulatory context". However, this delimitation affects the generalisability of the results. Bambra et al. (2014), who limited their review to the health systems of 15 high-income countries, concluded that since the majority of the studies related to interventions in the US system "[t]heir applicability to the UK and wider European health care context is highly questionable" (Bambra et al., 2014 p. 468). Hence, the decision on how the study is to be delimited becomes a balance between the need for context specificity and the strive to be able to generalise the findings. This overview shows that context-specific SRs are common in evaluation of public policy. This result contrasts from the study reported by Hannes and Harden (2011), where only a few of the qualitative SRs that they identified had focused solely on one particular country or national system. It further suggests that context has a particular meaning when investigating public policy; it is often appropriate to focus on a single policy, rather than comparing similar policies implemented in different countries. 5. Concluding discussion This overview identified 26 SRs of public policy and governance. These included a number of research areas, largely dominated by health and environment/natural resource management. Four of the SRs applied quantitative synthesis methods, two applied qualitative synthesis, and the majority (20 out of 26) of the identified SRs only narratively summarised the included studies. Therefore, this overview points to the considerable difficulty of quantitatively synthesising data from different studies of public policy. In order for meta-analysis to become possible and appropriate, the data must be sufficiently homogeneous. The multitude of study designs and ways of investigating a phenomenon taken by different researchers makes it difficult to compare and synthesise them into a coherent whole. This increases the risk of inappropriate comparisons and incorrectly drawn conclusions. The overview shows that narrative summaries can provide useful insights into how a topic has generally been studied, and display common categories or themes in the literature. However, it is worth discussing the quality of the narrative summaries. One of the main issues that sparked the methodological development of SR methods was the critique that literature reviews often lack scientific quality. The criticism concerned that “confronted with a large body of evidence, reviewers tend to focus on a small sub-set of studies but not to describe how they selected them; to be biased by their own perspective and findings in a particular field; and fail to assess the quality of studies or combine them appropriately” (Dixon-Woods et al., 2006a p. 29). While all SR included here have done systematic searches, in a narrative summary, all conclusions that go beyond describing the literature are at risk of making erroneous conclusions. This is because the quality of the included studies has not always been taken into account. In addition, the risk is that the reviewer draw conclusion about the overall evidence base using only the reviewers own reading of the literature without an appropriate method of synthesis. In order to make more grounded conclusions, established synthesis methods, such as content analysis or thematic analysis, would be preferred. To conclude, SR includes a group of methods that can be used to assess and evaluate environmental public policy. This overview has found examples of such reviews, as well as of policy evaluations that use SR methods in ways that would make them applicable to environmental policy evaluation. By looking at how these have been applied in other fields of policy it is evident that the methods can be used to estimate quantitative effects of a policy, gain a deeper understanding of barriers affecting the functioning of a policy or provide a description of how a policy has been researched. Three general lessons can be drawn from studying how the two

Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

51

Environmental Science and Policy 105 (2020) 47–55

M. Miljand

Acknowledgements

contract nr. 4 150501-18043. The author would like to thank the two anonymous reviewers whose comments helped strengthen this article.

This research was funded by Royal Swedish Academy of Sciences, Appendix A. Searches in Web of Science

Search string

Timespan

Index

Web of science categories

Results

Date

TOPIC: ("systematic review") AND TOPIC: (policy) TOPIC: ("systematic review") AND TOPIC: (policy) TOPIC: ("systematic review") AND TOPIC: (policy) TOPIC: ("systematic review") AND TOPIC: (policy)

All years

SCI-EXPANDED, SSCI, A& HCI, CPCI-S, CPCI-SSH, ESCI. SCI-EXPANDED, SSCI, A& HCI, CPCI-S, CPCI-SSH, ESCI. SCI-EXPANDED, SSCI, A& HCI, CPCI-S, CPCI-SSH, ESCI. SCI-EXPANDED, SSCI, A& HCI, CPCI-S, CPCI-SSH, ESCI.

Health policy services

271

15.11.2016

Nursing

107

15.11.2016

Social sciences interdisciplinary Education scientific disciplines Public administration Social work Education educational research Social issues Economics Social sciences biomedical Transportation Education special Political science Environmental studies Psychology multidisciplinary Ecology Computer science information systems Water resources Family studies Toxicology Psychology developmental Management Mathematical computational biology Food science technology Physiology Business Geography Planning development Ethics Pediatrics Rehabilitation Psychology Reproductive biology Industrial relations labor Information science library science Green sustainable science technology Psychology educational Biodiversity conservation Ophthalmology Primary health care Meteorology atmospheric sciences Medical ethics Substance abuse Urban studies Sociology Sport sciences Psychology applied Hospitality leisure sport tourism Engineering environmental Computer science theory methods Criminology penology Biology Psychology clinical Computer science software engineering Gerontology Energy fuels Agriculture multidisciplinary

365

15.11.2016

702

17.1.2017

All years

All years

All years

1445

Appendix B. Included reviews

Author

Aim

Aitken et al. (2015)

Examine the association between paid maternity leave and maternal health.

Auld et al. (2014)

Conclusion

The results suggested that paid maternity leave provided maternal health benefits, although this varied depending on the length of leave. Characteristics consistently associated with Assess whether policy innovations that more positive outcomes were built-in flexpromote the development and use of lowibility, defined time frames, and expendicarbon technologies have lasting consequences, and how characteristics of policies ture instruments. However, flexibility mechanisms were not always a guarantee of effectiveness and accountability implicasuccess and how longer time frames can tions.

52

Synthesis method

Context

Narrative summary

No limitations

Narrative summary

No limitations

Environmental Science and Policy 105 (2020) 47–55

M. Miljand present the risk of “locking’’ a country on particular policy pathways. Certain policy designs led to trade-offs across the different outcome criteria. Bambra et al. Assess the effectiveness of the UK’s welfare There is evidence that the various welfareto-work programmes operating in the UK in (2005) to work programmes for people with a the 1990s helped people with disabilities disability or chronic illness. into work who were previously on benefits. It is harder to determine, how much of the employment gained could be attributed to the specific welfare-to-work programme in which they participated. Bambra et al. Aim to assess whether it matters for equity Private insurance and out-of-pocket pay(2014) how you organize and pay for health care. ments as well as the marketization and privatization of services have either negative or inconclusive effects on health equity both in terms of access relative to need and in terms of health outcomes. Brisbois and de Aim to examine the extent to which litera- Many authors of studies relating to collaLoe (2016) ture discussing collaborative approaches to borative approaches to water governance are addressing power-related concerns in water governance reflects understanding and awareness of power-related considera- their work, and are aware of the importance of power for these processes. However, gaps tions. remain related to hidden structural and discursive power. Inconclusive results. The synthesis of results Cirera et al. (- Assessing the impact of trade policy, in 2014) terms of tariff reduction, on employment in suggests that the effects of tariff reductions on employment are country and trade developing countries. policy specific. In-person license renewal requirements Dugan et al. (- Assess state policies concerning older dri2013) vers and which policies appear to work to were associated with reduced risk for fatal crashes. Restricted licenses were associated reduce older driver crashes. with reduced number of miles driven per week. More intensive renewal requirements and being the subject of a medical report to the licensing authority was associated with delicensure. Found clear evidence that the prospect of Filges et al. (2- Study the impact of exhaustion of unemexhaustion of benefits results in a signifi015) ployment benefits on the exit rate out of unemployment and into employment prior cantly increased incentive for finding work. to benefit exhaustion or shortly thereafter. Anti-bullying policies might be effective at Hall (2017) The aim of the review is to evaluate the reducing bullying if their content is based methodological characteristics and summarizes substantive findings of studies ex- on evidence and sound theory and if they amining the effectiveness of school bullying are implemented with a high level of fidelity. policies Howes et al. (- The aim of the review is to understand the Three key factors were recurring in the included articles. First, there are ongoing 2017) implementation failure of environmental economic incentives (supported by policy) sustainability. Mainly why the implementation of sustainability policies has contrib- for private and public activities to continue to exploit natural resources without deeper uted to the failure to arrest the overall decline in environmental quality. A range of consideration of the associated damage to the environment. Second, governments causes, implementation traps, and knowleither do not have the capacity and/or the edge/scope issues is identified and anapolitical will to implement effective suslysed. tainability policies. Third, the seriousness of sustainability issues and the urgent need for change have not been effectively communicated to key stakeholders. In addition, developing countries will face additional challenges with regards to the capacity of their public sector to catalyse change. Inconclusive results. Evidence on the relaHyde and Sho- Examining the influence of organizational tionship of public health organization, perrtell (201- characteristics on public health perforformance, and health outcomes is limited. mance and health status. 2) Comprehensive strategies involving mulHyseni et al. (- The aim is to assess whether upstream tiple components (reformulation, food la2017) interventions (structural interventions belling and media campaigns) and "uptaking place at the population level and typically involve policies such as regulatory stream" population-wide policies such as mandatory reformulation generally appear approaches, taxes or subsidies) have a to achieve larger reductions in populationhigher effectiveness compared to downstream intervention (individual level inter- wide salt consumption than "downstream", individually focussed interventions. ventions) to reduce salt intake. Tenure security is important, in addition to Lawry et al. (- Assess the impacts of interventions to 2014) strengthen land property rights on agricul- being a pre-condition to farm investments tural and livelihood outcomes in rural areas that foster productivity and increase farm incomes, growing investor interest in farmof low and middle income countries. land as well as contextual changes– population growth, changing settlement patterns, political conflict, environmental

53

Narrative summary

One country: United Kingdom

Limitation to high, middle and/or low Narrative summary A review of systematic reviews. income countries: 15 high-income countries The results were synthesized by intervention type.

Narrative summary Systematic map

No limitations

Meta-analysis

Limitation to high, middle and/or low income countries: developing countries

Narrative summary

One country: United States

Meta-analysis

No limitations

Narrative summary, narrative thematic synthesis.

No limitations

Narrative summary

No limitations

Narrative summary Common themes across studies were presented. Narrative summary

One country: United states

Bivariate meta-regressions and an aggregative metasummary

Limitation to high, middle and/or low income countries: developing countries.

No limitations

Environmental Science and Policy 105 (2020) 47–55

M. Miljand degradation and climate change– are among the factors underscoring the need to better secure tenure rights in developing countries. Lwasa (2015) Synthesizes literature on adaptation policy Adaptation is predominantly focused on and practice in river deltas located in Africa intra-delta vulnerabilities and is biased toward development sectors. These efforts are and South Asia and identifies future renot properly mainstreamed in national-level search priorities targeting climate change policies for resilience building and transadaptation in large delta regions. formational adaptation. Adaptation policy gaps exist surrounding knowledge advancement, scaling up actions, the “adaptation gap’’ and the move from incremental adaptation to transformational adaptation. Macura et al. Assessing the impact of governance type on Inconclusive results. The presented results call attention to the research gaps in the (2015) the conservation effectiveness of forest field of conservation governance. While protected areas. conducting analysis of conservation intervention effects, research has to account for and report governance variables. McPhedran (2- Assess the impacts of legislative reform on No study found statistical evidence of any 016) firearm homicide in Australia. significant impact of the legislative changes on firearm homicide rates. The review find evidence to support that O’Reilly et al. The aim is to evaluate the effect of HIV some types of HIV prevention policy inter(2017) policy interventions on changing HIV-related behaviors in low and middle income ventions in low-and middle-income countries can be an effective strategy to change countries. HIV-related behaviours. Advance our understanding for the extent to Articles were found that indicate that economic incentives can impact on intrinsic which the use of economic incentives can motivations for engaging in biodiversity or undermine (crowed out) or reinfores peoecosystem service conservation. ple's intrinsic motivations to engage in bodiversity or ecosystem conservation. Inconclusive results. Even though we Meng et al. (2- Assess strategies that have been used to 011) expand or improve health insurance plans cannot make generic conclusions about the policy implications of strategies in different or community-based health insurance settings, policy making in expanding health schemes to vulnerable groups. insurance coverage in one context could be stimulated by practice in other contexts. Meurk et al. (- Assess the e-mental health service use for Little research focuses explicitly on policy 2016) depressive and anxiety disorders development and implementation planning; most research provides an e-services perspective. Peñaloza et al. Assess interventions to reduce emigration of Only one study was identified. The only evidence found was from an intervention in (2011) health care professionals from low and a high income country that increased the middle income countries movement of health professionals from a low and middle income country. Perkins et al. Assess the impact of partnership working on Little evidence that partnerships have produced better health outcomes for local/ (2010) public health, and whether these partnerships have delivered better health outcomes target populations or reduced health inequalities. for local/target populations To outline the current state of the literature The main governance challenges include Pittman and on the topic of governance across the land determining boundaries, addressing crossArmitage scale effects, and accessing knowledge. sea interface, and examine the predomi(2016) However, few empirical studies of `governnance of different approaches to address ance across the land-sea interface have been land-sea interactions, characterize how governance is conceptualized, and investi- completed. gate governance challenges. Samii et al. (2- Effects of decentralized forest management The study finds that DFM reduce defores014) (DFM) on deforestation and poverty in low tation rates. In terms of program effects on human welfare and poverty outcomes, the and middle income countries. evidence is limited and no conclusions can be drawn. When it comes to assessing the extent to which conservation and poverty reduction goals conflict not enough evidence exist to make a case for conservation and poverty reduction goals being complementary in DFM programming. Sud et al. (20- Review literature to understand governance There is ambiguity in goal setting among 4 15) mechanisms for climate adaptation in three out of 6 countries which may constrain policy implementation. Further, that mainmajor glacier-fed river systems in South tain the levels of political jurisdiction as the Asia. functional scale, raises a question of fit with the complex issues of climate change that play out in ecological scale. Varabyova and The efficiency of health care production in The meta-analysis revealed low correlations between country rankings suggesting a lack Muller (2- OECD countries. of internal validity of the efficiency esti016) mates. Methodological problems of existing

54

Narrative summary Thematic focus of the literature reviewed

Limitations connected to rivers/deltas: river deltas located in Africa and South Asia

Narrative summary Systematic map

No limitations

Narrative summary A statistical overview

One country: Australia

Narrative summary

Limitation to high, middle and/or low income countries: low and middle income countries.

Narrative summary

No limitations

Narrative summary

No limitations

Narrative summary

One country: Australia

Narrative summary Aimed to do a meta-analysis but as only one study was identified no synthesis was possible. Qualitative synthesis Informed by a critical interpretive synthesis (CIS) approach. Qualitative synthesis, including grounded theory and network analysis.

Limitation to high, middle and/or low income countries: low and middle income countries

Narrative summary

Limitation to high, middle and/or low income countries: low and middle income countries

Narrative summary A systematic map

Limitations connected to rivers/deltas: three Himalayan river basins: Brahmaputra, Ganges and Indus (spread over six countries: Afghanistan, Bangladesh, Bhutan, India, Nepal and Pakistan).

Meta-analysis

Limitation to high, middle and/or low income countries: OECD member countries

One country: United Kingdom

No limitations

Environmental Science and Policy 105 (2020) 47–55

M. Miljand cross-country comparisons of the efficiency of health care systems draw into question the ability of these comparisons to provide meaningful guidance to policy-makers.

Lawry, S., Samii, C., Hall, R., Leopold, A., Hornby, D., Mtero, F., 2014. The Impact of Land Property Rights Interventions on Investment and Agricultural Productivity in Developing Countries: a Systematic Review. Lwasa, S., 2015. A systematic review of research on climate change adaptation policy and practice in Africa and South Asia deltas. Reg. Environ. Change 15, 815–824. Macura, B., Secco, L., Pullin, A.S., 2015. What evidence exists on the impact of governance type on the conservation effectiveness of forest protected areas? Knowledge base and evidence gaps. Environ. Evid. 4, 1. Macura, B., Suškevičs, M., Garside, R., Hannes, K., Rees, R., Rodela, R., 2019. Systematic reviews of qualitative evidence for environmental policy and management: an overview of different methodological options. Environ. Evid. 8, 24. Mays, N., Roberts, E., Popay, J., 2001. Synthesising Research Evidence. Studying the Organisation and Delivery of Health Services: Research Methods. pp. 220. Mcphedran, S., 2016. A systematic review of quantitative evidence about the impacts of Australian legislative reform on firearm homicide. Aggress. Violent Behav. 28, 64–72. Meng, Q.Y., Yuan, B.B., Jia, L.Y., Wang, J.A., Yu, B.R., Gao, J., Garner, P., 2011. Expanding health insurance coverage in vulnerable groups: a systematic review of options. Health Policy Plan. 26, 93–104. Meurk, C., Leung, J., Hall, W., Head, B.W., Whiteford, H., 2016. Establishing and governing e-Mental health care in Australia: a systematic review of challenges and a call for policy-focussed research. J. Med. Internet Res. 18. Mickwitz, P., 2003. A framework for evaluating environmental policy instruments context and key concepts. Evaluation 9, 415–436. Mickwitz, P., 2006. Environmental Policy Evaluation: Concepts and Practice. Suomen Tiedeseura. Noblit, G.W., Hare, R.D., 1988. Meta-ethnography: Synthesizing Qualitative Studies. Sage. O’reilly, K.R., D’aquila, E., Fonner, V., Kennedy, C., Sweat, M., 2017. Can policy interventions affect HIV-Related behaviors? A systematic review of the evidence from lowand middle-income countries. AIDS Behav. 21, 626–642. Overman, S., 2016. Great expectations of public service delegation a systematic review. Public Manag. Rev. 18, 1238–1262. Pawson, R., 2006. Evidence-Based Policy: a Realist Perspective. Sage. Pawson, R., Greenhalgh, T., Harvey, G., Walshe, K., 2004. Realist synthesis: an introduction. ESRC Research Methods Programme. University of Manchester, Manchester. Pawson, R., Greenhalgh, T., Harvey, G., Walshe, K., 2005. Realist review–a new method of systematic review designed for complex policy interventions. J. Health Serv. Res. Policy 10, 21–34. Peñaloza, B., Pantoja, T., Bastías, G., Herrera, C.A., Rada, G., 2011. Interventions to reduce emigration of health care professionals from low- and middle-income countries. Cochrane Database Syst. Rev. https://doi.org/10.1002/14651858.CD007673.pub2/ abstract. [Online]. Available:. Perkins, N., Smith, K., Hunter, D.J., Bambra, C., Joyce, K., 2010. ’What counts is what works’? New Labour and partnerships in public health. Policy Polit. 38, 101–117. Pittman, J., Armitage, D., 2016. Governance across the land-sea interface: a systematic review. Environ. Sci. Policy 64, 9–17. Pullin, A.S., Knight, T.M., 2003. Support for decision making in conservation practice: an evidence-based approach. J. Nat. Conserv. 11, 83–90. Pullin, A.S., Knight, T.M., 2005. Assessing conservation management’s evidence base: a survey of management-plan compilers in the United Kingdom and Australia. Conserv. Biol. 19, 1989–1996. Pullin, A.S., Knight, T.M., Stone, D.A., Charman, K., 2004. Do conservation managers use scientific evidence to support their decision-making? Biol. Conserv. 119, 245–252. Pullin, A.S., Knight, T.M., Watkinson, A.R., 2009. Linking reductionist science and holistic policy using systematic reviews: unpacking environmental policy questions to construct an evidence‐based framework. J. Appl. Ecol. 46, 970–975. Rode, J., Gomez-Baggethun, E., Krause, T., 2015. Motivation crowding by economic incentives in conservation policy: a review of the empirical evidence. Ecol. Econ. 117, 270–282. Samii, C., Paler, L., Chavis, L., Kulkarni, P., Lisiecki, M., 2014. Effects of Decentralized Forest Management (DFM) on Deforestation and Poverty in Low and Middle Income Countries: A Systematic Review. Sandelowski, M., Barroso, J., Voils, C.I., 2007. Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Research in nursing & health 30 (1), 99–111. Sud, R., Mishra, A., Varma, N., Bhadwal, S., 2015. Adaptation policy and practice in densely populated glacier-fed river basins of South Asia: a systematic review. Reg. Environ. Change 15, 825–836. Thomas, J., Harden, A., 2008. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC medical research methodology 8 (1), 45. Varabyova, Y., Muller, J.M., 2016. The efficiency of health care production in OECD countries: a systematic review and meta-analysis of cross-country comparisons. Health Policy 120, 252–263. Weber, M., Driessen, P.P., Runhaar, H.A., 2014. Evaluating environmental policy instruments mixes; a methodology illustrated by noise policy in the Netherlands. J. Environ. Plan. Manag. 57, 1381–1397. Wolfswinkel, J.F., Furtmueller, E., Wilderom, C.P., 2013. Using grounded theory as a method for rigorously reviewing literature. Eur. J. Inf. Syst. 22, 45–55. Wyborn, C., Louder, E., Harrison, J., Montambault, J., Montana, J., Ryan, M., Bednarek, A., Nesshöver, C., Pullin, A., Reed, M., 2018. Understanding the impacts of research synthesis. Environ. Sci. Policy 86, 72–84.

References Aitken, Z., Garrett, C.C., Hewitt, B., Keogh, L., Hocking, J.S., Kavanagh, A.M., 2015. The maternal health outcomes of paid maternity leave: a systematic review. Soc. Sci. Med. 130, 32–41. Auld, G., Mallett, A., Burlica, B., Nolan-Poupart, F., Slater, R., 2014. Evaluating the effects of policy innovations: lessons from a systematic review of policies promoting lowcarbon technology. Glob. Environ. Change Hum. Policy Dimen. 29, 444–458. Bambra, C., Garthwaite, K., Hunter, D., 2014. All things being equal: does it matter for equity how you organize and pay for health care? A review of the international evidence. Int. J. Health Serv. 44, 457–477. Bambra, C., Whitehead, M., Hamilton, V., 2005. Does’ welfare-to-work’ work? A systematic review of the effectiveness of the UK’s welfare-to-work programmes for people with a disability or chronic illness. Soc. Sci. Med. 60, 1905–1918. Barnett-Page, E., Thomas, J., 2009. Methods for the synthesis of qualitative research: a critical review. BMC Med. Res. Methodol. 9, 59. Bilotta, G.S., Milner, A.M., Boyd, I., 2014. On the use of systematic reviews to inform environmental policies. Environ. Sci. Policy 42, 67–77. Booth, A., Mshelia, S., Analo, C.V., Nyakang’o, S.B., 2019. Qualitative evidence syntheses: assessing the relative contributions of multi‐context and single‐context reviews. J. Adv. Nurs. 75 (12), 3812–3822. Brisbois, M.C., De Loe, R.C., 2016. Power in collaborative approaches to governance for water: a systematic review. Soc. Nat. Resour. 29, 775–790. Campbell, R., Pound, P., Morgan, M., Daker-White, G., Britten, N., Pill, R., Yardley, L., Pope, C., Donovan, J., 2012. Evaluating Meta Ethnography: Systematic Analysis and Synthesis of Qualitative Research. Christiansen, P.M., 1996. Governing the Environment: Politics, Policy, and Organization in the Nordic Countries, Nordic. Cirera, X., Willenbockel, D., Lakshman, R.W.D., 2014. Evidence on the impact of tariff reductions on employment in developing countries: a systematic review. J. Econ. Surv. 28, 449–471. Collaboration For Environmental Evidence, 2013. Guidelines for Systematic Review and Evidence Synthesis in Environmental Management. Version 4.2. Environmental Evidence. http://environmentalevidence.org/wp-content/uploads/2014/06/ Review-guidelinesversion4.2-finalPRINT.pdf. Dacombe, R., 2017. Systematic reviews in political science: what can the approach contribute to political research? Polit. Stud. Rev., 1478929916680641. Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., Sutton, A., 2005. Synthesising qualitative and quantitative evidence: a review of possible methods. J. Health Serv. Res. Policy 10, 45–53. Dixon-Woods, M., Bonas, S., Booth, A., Jones, D.R., Miller, T., Sutton, A.J., Shaw, R.L., Smith, J.A., Young, B., 2006a. How can systematic reviews incorporate qualitative research? A critical perspective. Qual. Res. 6, 27–44. Dixon-Woods, M., Cavers, D., Agarwal, S., Annandale, E., Arthur, A., Harvey, J., Hsu, R., Katbamna, S., Olsen, R., Smith, L., 2006b. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med. Res. Methodol. 6, 35. Dugan, E., Barton, K.N., Coyle, C., Lee, C.M., 2013. U.S. Policies to enhance older driver safety: a systematic review of the literature. J. Aging Soc. Policy 25, 335–352. Eaves, Y.D., 2001. A synthesis technique for grounded theory data analysis. J. Adv. Nurs. 35, 654–663. Filges, T., Geerdsen, L.P., Knudsen, A.S.D., Jorgensen, A.M.K., 2015. Unemployment benefit exhaustion: incentive effects on job-finding rates. Res. Soc. Work Pract. 25, 21–43. Forss, K., Marra, M., Schwartz, R., 2011. Evaluating the Complex: Attribution, Contribution, and Beyond. Transaction Publishers. Gough, D., Oliver, S., Thomas, J., 2012. An Introduction to Systematic Reviews. Sage. Grant, M.J., Booth, A., 2009. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf. Libr. J. 26, 91–108. Gysen, J., Bruyninckx, H., Bachus, K., 2006. The modus narrandi a methodology for evaluating effects of environmental policy. Evaluation 12, 95–118. Hall, W., 2017. The effectiveness of policy interventions for school bullying: a systematic review. J. Soc. Social Work Res. 8, 45–69. Hanberger, A., 2001. What is the policy problem? Methodological challenges in policy evaluation. Evaluation 7, 45–62. Hannes, K., Harden, A., 2011. Multi‐context versus context‐specific qualitative evidence syntheses: combining the best of both. Res. Synth. Methods 2, 271–278. Hannes, K., Lockwood, C., 2011. Synthesizing Qualitative Research: Choosing the Right Approach. John Wiley & Sons. Howes, M., Wortley, L., Potts, R., Dedekorkut-Howes, A., Serrao-Neumann, S., Davidson, J., Smith, T., Nunn, P., 2017. Environmental sustainability: a case of policy implementation failure? Sustainability 9. Hyde, J.K., Shortell, S.M., 2012. The structure and organization of local and state public health agencies in the US a systematic review. Am. J. Prev. Med. 42, S29–S41. Hyseni, L., Elliot-Green, A., Lloyd-Williams, F., Kypridemos, C., O’flaherty, M., Mcgill, R., Orton, L., Bromley, H., Cappuccio, F.P., Capewell, S., 2017. Systematic review of dietary salt reduction policies: evidence for an effectiveness hierarchy? PLoS One 12. Kearney, M.H., 2001. Enduring love: a grounded formal theory of women’s experience of domestic violence. Res. Nurs. Health 24, 270–282. Krogstrup, H.K., 2003. Evalueringsmodeller: evaluering på det sociale områdeevaluering på det sociale område. Systime Academic.

55