Developing a public health policy-research nexus: An evaluation of Nurse Practitioner models in aged care

Evaluation and Program Planning 40 (2013) 55–63

Contents lists available at SciVerse ScienceDirect

Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan

Developing a public health policy-research nexus: An evaluation of Nurse Practitioner models in aged care Brenton Prosser *, Shannon Clark, Rachel Davey, Rhian Parker Centre for Research and Action in Public Health, University of Canberra, Bruce, ACT 2606, Australia

A R T I C L E I N F O

A B S T R A C T

Article history: Received 29 October 2012 Received in revised form 10 May 2013 Accepted 27 May 2013

A frustration often expressed by researchers and policy-makers in public health is an apparent mismatch between respective priorities and expectations for research. Academics bemoan an oversimplification of their work, a reticence for independent critique and the constant pressure to pursue evaluation funding. Meanwhile, policy-makers look for research reports written in plain language with clear application, which are attuned to current policy settings and produced quickly. In a context where there are calls in western nations for evidence based policy with stronger links to academic research, such a mismatch can present significant challenges to policy program evaluation. The purpose of this paper is to present one attempt to overcome these challenges. Specifically, the paper describes the development of a conceptual framework for a large-scale, multifaceted evaluation of an Australian Government health initiative to expand Nurse Practitioner models of practice in aged care service delivery. In doing so, the paper provides a brief review of key points for the facilitation of a strong research-policy nexus in public health evaluations, as well as describes how this particular evaluation embodies these key points. As such, the paper presents an evaluation approach which may be adopted and adapted by others undertaking public health policy program evaluations. ß 2013 Elsevier Ltd. All rights reserved.

Keywords: Research-policy nexus Policy evaluation Public health Nurse Practitioner

1. Introduction The relationship between academic research and public policy is changing. Ever faster news media cycles constrain research input into conventional policy review cycles. Increased emphasis on value for money within taxpayer-funded programs requires evaluations to be more attuned to the efficiency and financial viability of policy initiatives. Growing interest in evidence-based policy in response to complex social problems demands that findings be relevant and more accessible to policy-makers. Together, these trends present new challenges which require new approaches to developing a nexus between research and policy within policy program evaluations. This paper takes up the challenge of developing an evaluation that is based on a theoretical framework through five distinct steps. First, a brief summary of the policy history that underlies the establishment and support of Nurse Practitioners in aged care within Australia is presented. This is followed by an overview of the key objectives for a current evaluation of an Australian

* Corresponding author at: c/- Centre for Research and Action in Public Health, Building 22, University of Canberra, Bruce, ACT 2606, Australia. Tel.: +61 2 62012914. E-mail address: [email protected] (B. Prosser). 0149-7189/$ – see front matter ß 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.evalprogplan.2013.05.003

Government policy program in this space. Third, the paper details a theoretical approach that expands this evaluation beyond the assessment of program outcomes against objectives through a three-strand conceptual framework that provides a foundation for rigorous analysis. Finally, the evaluation is considered in the light of literature considering the development of a research-policy nexus in health. Due to this focus on describing an evaluation approach that responds to research-policy challenges identified in the literature, a detailed examination of the findings of the evaluation in relation to accessibility of health services and clinical practice is necessarily beyond the scope of this paper. However, what the paper does usefully present is an evaluation approach as an example of a careful synthesis of often conflicting policy and research priorities, while providing sufficient detail for the approach to be considered for use by others conducting public health policy evaluations. 2. Nurse Practitioners in aged care: the Australian policy context Before considering the conceptual framework of this evaluation approach, it is important to take a moment to define the Nurse Practitioner role, as it differs between nations (Sibbald, Laurant, & Reeves 2006). In Australia, the role of Nurse Practitioner (NP) is defined as a Registered Nurse who is educated at Masters level and

56

B. Prosser et al. / Evaluation and Program Planning 40 (2013) 55–63

endorsed to function autonomously and collaboratively in an advanced and extended clinical role. The role may include (but is not limited to), assessment and management of health needs, the direct referral of patients to other health care professionals, prescribing medications and ordering diagnostic investigations within their scope of practice (Australian Nursing and Midwifery Council, 2012). The educational requirements and scope of practice of the role are shaped by the context and/or country, which constrain direct comparison of research. That said, there is some evidence in international contexts that the introduction of NPs increases service-users’ satisfaction, improve health outcomes, reduce the prescription of pharmaceuticals, and decrease readmission to acute care (Brown & Grimes, 1995; Horrocks, Anderson, & Salisbury, 2002; Laurant, Hermens, Braspenning, Sibbald, & Grol, 2004; Laurant et al., 2008; Mundinger et al., 2000). Further, service users have been shown to be accepting of NPs as care providers in Australia (Parker, Forrest, Ward, McCracken, Cox, & Derrett, 2013) and North America (Agosta, 2009), while there is some evidence of the positive impact of NPs in aged care settings (Christian & Baker, 2009; Dick & Frazier, 2006). However, there remains a lack of evidence in relation to the impact of the introduction of NPs in the Australian context. Medical workforce shortages have been a cause for political concern in Australia since the mid-1990s, and widely acknowledged that shortages also existed within most other health professions. The Australian Productivity Commission’s (2005) report, Economic Implications for an Ageing Australia, argued that a new national approach was needed to deal with structural pressures on the health system (caused by rising expectations of patients, the ageing of the population and a declining health workforce). The Commission suggested that one approach would be to integrate new models of care and workforce practices. This approach was adopted by the Australian Federal Government after 2007 through a greater emphasis on primary and community health care (Baum, 2009), a change in health care models, and a shift in role boundaries for health professionals (Jolly, 2009). Subsequently, the Government introduced legislation (Jolly, 2009) that would allow eligible NPs in private practice in Australia to access the Medicare Benefits Scheme (MBS) and Pharmaceutical Benefits Scheme (PBS). These schemes provide subsidised access to selected health services and pharmaceuticals, making these services more affordable to Australian citizens, particularly those on lower incomes or government pensions. The passage of this legislation in 2010 established a situation where eligible NPs could potentially provide more accessible and cost-effective delivery of health services in home and community (rather than clinic or hospital) settings. As was noted earlier, international definitions vary for nurse practitioners, which make direct comparisons difficult, other than to say that the role has grown internationally in recent years (Auerbach, 2012; Donald et al., 2010; Sibbald et al., 2006). However in Australia in recent years, the increased government support described above has been reflected by Nurse Practitioner numbers now amounting to over seven hundred (Nursing and Midwifery Board of Australia, 2012), with over one hundred specialising in aged care service delivery. It was in this emerging context that a national evaluation of NP models in aged care was announced by the Australian Government as part of the 2010–2011 Federal Budget.

approximately thirty models were selected by the Australian Government to be included in the NP Program. The models funded under the NP Program aim to: 1. improve access to primary health care for clients of residential and community aged care services; 2. demonstrate effective, economically viable and sustainable models of practice; and 3. facilitate the growth of the aged care Nurse Practitioner workforce. The Australian Government initiative also includes an independent national evaluation (NP Evaluation) of the Program between 2012 and 2014. The primary outcome of this NP Evaluation is to assess the extent to which the NP Program and each of the models is meeting the aims listed above. The successful proposal for the NP Evaluation proposed to: 1. compare local Nurse Practitioner models against the NP Program aims; 2. assess resource requirements and costs; 3. analyse effectiveness including critical success factors and challenges; 4. explore consumer and provider experiences of delivery models; and 5. assess viability and sustainability of Nurse Practitioner models. The evaluation will be conducted by academics from the University of Canberra, the Australian National University and the Australian Institute of Health and Welfare. The oversight of the evaluation will be through a Project Steering Committee, which includes representatives of the Commonwealth Department of Health and Ageing and the evaluation team. 4. Conceptual framework The NP Evaluation assesses a policy program that aims to address an issue of national significance – improving access to affordable primary health care for the ageing population. Generating an evidence base for the evaluation, the models of care within it, and the effectiveness, viability, and sustainability of the NP role in aged care requires an approach which is theoretically informed and methodologically rigorous. However, the NP Evaluation also seeks to respond to calls for new conceptual frameworks that embrace research and policy priorities, as well as show more theoretical rigour (Petticrew, Whitehead, Macintrye, Graham, & Egan, 2004). For the purposes of this paper, the term ‘conceptual framework’ is used to refer to organising principles that give coherence to and connect the diverse and inherently complex aspects of the evaluation (Meyer, Davis, & Mays, 2012). Drawing from the work of Smyth (2004), we identify the key components of such frameworks to be:

3. About the NP Evaluation

 epistemologically clear ideas, principles and theories to underpin rigorous enquiry;  systematically clear approaches to scaffold data collection/ analysis/reflection; and  procedurally clear structures for discussing and presenting findings.

The $18.7 m Nurse Practitioner – Aged Care Models of Practice Program (NP Program) is an initiative of the Australian Government to further grow the number of NPs providing aged care services (Department of Health and Ageing, 2011). In 2011, applications were invited for aged care models across Australia, from which

Such an approach requires flexibility to cope with the practicalities of the real-world, a far cry from the controlled confines of the ‘gold-standard’ of the random controlled trial. Additionally, the findings need to be relevant and accessible to policy-makers and stakeholders.

B. Prosser et al. / Evaluation and Program Planning 40 (2013) 55–63

A brief overview of challenges surrounding the research-policy gap from public health literature will be provided later in the discussion section of this paper. However, from this discussion four key themes have been identified around which the conceptual framework for the NP Evaluation were developed, these include:  closer links between policy priorities and research evidence;  research evidence that is more accessible and relevant to policymakers;  more collaborative relationships and constant communication between researchers and policy-makers; and  the use of mixed methods, the production of local cases and inclusion of multiple stakeholder perspectives. The theoretical and methodological approaches that aim to satisfy these often competing demands are outlined below. 4.1. Theoretical approach A drive for increased patient autonomy and community participation, as well as decreased institutionalism and cost, have shifted the balance in western nations regarding who is seen as responsible for health care. Australia has not been immune to these changes (Dow & McDonald, 2007; Duckett, 2007). Such changes alter the structure and nature of health care work and are creating unprecedented challenges for the provision of health care for the ageing Australian population (National Health Workforce Taskforce, 2009). Internationally, attempts to respond to these issues have resulted in changing models of health care and shifting professional role boundaries, including the development of extended roles for nursing (e.g., the role of the NP). Given this context, policy program evaluations require more than a description of performance of NP models against program aims. What is also required is a theoretical conceptualisation of the cultural, economic, policy and professional drivers that have influenced the emergence of NP models, as well as their implications for the development of new approaches to aged care service delivery. By necessity, this is ‘real-world’ research in which variables are beyond the control of the research team and where the unique is the norm. What is needed is an evaluation approach that is attuned to macro, meso and micro levels, as well as policy and professional implications. As Botterill and Hindmoor (2012) argue, policy emerges through a messy process, but this process should not be seen as the only constraint to evidence-based policy. Rather, the common evaluation approach of systematically collecting and communicating evidence for policy-makers in relation to meeting policy objectives should also be seen as constraining and distorting of evidence. Further, evaluation approaches should be sensitive to the confusion around definitions of health service delivery ‘models’ (Davidson, Halcomb, Hickman, Phillips, & Graham, 2006). For instance, the NP Program uses the term model as a generic description for each of the thirty different organisational arrangements that are receiving federal funding to provide NP aged care services in their community. However, the evaluation framework uses a more specific definition of model as a set of arrangements that provide service user orientated services across clinical and community settings while aiming to provide a continuity of care within a broader philosophy of nursing (McCabe & Jacobs, 2012). In this sense, models are both different sets of arrangements that fund a NP role and thirty different configurations of NPs using their scopes of practice. With this in mind, the proposed NP Evaluation objectives and theoretical approach are orientated to theoretical concerns, contextual influences, situational complexity, professional capacity and the need for stakeholder collaboration.

57

To achieve these purposes, a Realist Evaluation (Pawson & Tilley, 2004, 1997) perspective will be adopted for this evaluation. This is a theory-driven perspective which includes an emphasis on the importance of context and the use of multiple methods (Rycroft-Malone, Fontela, Bick, & Seers, 2010). Realistic Evaluation is concerned with the identification of underlying mechanisms, and because these mechanisms are located in complex contexts, it is important to understand the relationships between operationalisation and outcome. What underlies the Realistic Evaluation perspective is the contention that all social programs are products of human intentions and actions shaped by complex contexts (Pawson & Tilley, 2004). Hence, programs, like the NP Program are borne out of reasoning how things could be achieved, done differently, improved, and/or strengthened. These ideas and intentions are interpreted into policy by policy-makers, and subsequently passed on to ‘on the ground’ practitioners to enact, which (hopefully) results in improvements of service user experiences. Whether aware of it or not, participants in each stage of this process are informed by conceptualisations, which shape their interpretation, action and implementation. These conceptualisations may be related to culture, human betterment, economic demand, social conditions, population trends or future perceived health crises. In this perspective, when one evaluates a program, it is important to consider the key ideas that justify its existence, as well as understand its stakeholders and contextual influences. This evaluation approach does not just ask if a program works, but also asks what works for whom, when, in what circumstances, and why? Put simply, such a perspective can look into why a delivery model that is internally effective and efficient may not be viable or sustainable due to external policy, population, socio-cultural and economic factors. However, there are also challenges to conducting Realistic Evaluation with diverse cases in different locales and producing concrete and evidence based assertions about context and outcome. That is why, as will be discussed below, a case study approach that groups cases by common features will used to reduce the influence of specific contexts on outcomes, and why a linkage with national administrative data bases will be conducted to identify patterns in health service use both before and after the introduction of NP services. 4.2. A conceptual frame The Realistic Evaluation perspective provides a theoretical foundation on which we have constructed a conceptual frame that supports a mixed method evaluation approach. Such an approach is sensitive to the diversity of models within the NP Program and the complexity of contexts in which these models operate. Further, a mixed method evaluation approach enables data collection in relation to the broad brief of the NP Evaluation, which includes quantitative data on access to health services and the economic viability of delivery models, as well as qualitative data on the critical factors in the effectiveness of these models and different accounts of stakeholder experiences. For the purposes of clarity in this paper, the three methodological strands within the NP Evaluation approach are labelled as ‘qualitative’, ‘case study’ and ‘quantitative’ (Fig. 1). Within a mixed method approach, each data collection method has its own limitations. However, the use of several data sources in triangulation can help overcome these limitations by providing rich and multiple perspectives on the same phenomena. Further, when used independently (according to rigorous methodological standards), findings from different methodological approaches that reinforce each other can be used to identify higher order of reliability. In addition, to enhance the robustness of the data, the NP Evaluation team will also use a reflective and iterative approach

B. Prosser et al. / Evaluation and Program Planning 40 (2013) 55–63

58

• • • • •

• • •

• • •

• • •

• • • • Fig. 1. NP Evaluation conceptual framework.

to checking findings and observations from across these methodological approaches (this includes feeding back findings to NP models as formative input during the evaluation). 4.2.1. The qualitative strand The practical workings of each model will be conducted by adapting the RE-AIM approach (Glasgow, Vogt, & Boles, 1999) which is often used for evaluation of health interventions. Glasgow and colleagues argue that too often health evaluations have considered only one or two dimensions of quality and overlooked the broader public health significance of interventions. Additionally, they argue that the traditional scientific research paradigm in which experimental method is dominant is not predictive of how successful an intervention will be in the real-world. In response, the RE-AIM model examines interventions by considering the reach of programs, the representativeness of participants in settings, the effectiveness and impact of a health intervention, as well as the capacity for health interventions to maintain these impact over the longer term. The central tenet of the RE-AIM model is that it relies on the combined impact of five evaluative dimensions. These dimensions are: Reach, Efficacy, Adoption, Implementation, and Maintenance (Glasgow, Klesges, Dzewaltowski, Estabrooks, & Vogt, 2006). Importantly, the NP Evaluation will not attempt to quantify each of the RE-AIM domains (McKenzie, Naccarella, Stewart, & Thompson, 2007), but rather it adopts the model as a primarily qualitative, inductive and descriptive approach. It will use the five domains to provide a systematic structure for evaluating the NP model of practice in each of the selected sites, while also enabling a

number of data sources to be incorporated to build a detailed, contextualised picture of each selected site. It should be noted that not all models (or sites within models) will be selected for analysis through the RE-AIM framework. While all sites will participate in quantitative and case study analysis, the qualitative aspect of the study selects sites based on the principle of capturing an illustrative (rather than representative), sample of models participating in the program. That is, sites will be selected to capture the diversity of the types of organisations, the location, and the target populations the models serve. It is anticipated that sixteen of the approximately thirty models will be selected for consideration and reports for each site will be organised around the five RE-AIM domains. Based on initial document analysis of tender applications, the models will be stratified according to their organisation type, their location, and whether they are targeting a specific condition or population. The first consideration is the type of organisation delivering the model. There are two broad categories of organisations participating in the program: (a) not-for-profit residential aged care providers; and (b) community-based health care providers. The second consideration is location, where the Australian Standard Geographical Classification - Remoteness Area scale will be used to break down each of the organisation types into subcategories of ‘major cities’ or ‘regional/rural’. In keeping with the principle of capturing an illustrative range of models, models regarded as ‘revelatory’ or ‘illustrative’ will also be selected, this includes unique specialist services, unique

B. Prosser et al. / Evaluation and Program Planning 40 (2013) 55–63

demographic components or culturally unique populations. The final consideration in site selection is whether the site has an endorsed NP in place. Applying these categories successively produces a filter for selecting sites. Drawing on interviews, surveys and site audits, the reports will:  provide practical, grounded descriptions of each selected model, its key features, and how it works in its local context;  uncover and explicate ways that NP models are managed and accomplished on a day-to-day basis; and  identify different stakeholders’ understandings and experiences of the model, and successes and shortcomings, particularly in relation to the RE-AIM domains. By considering the interplay between the specific features of a model (i.e., the NP, the organisation, the location, specialisation and clientele, as well as the experiences of stakeholders), these reports will enable comparisons and differentiations to be made between different models and their specific features. The intent is to generate information of relevance to policy-makers and practitioners that will not necessarily be identified through deductive approaches. 4.2.2. The quantitative strand The broader impact of models across the NP Program will be assessed through collection of data on client consultations, medication use and patterns in health service use. While it is important to provide description of what works for particular models in particular contexts, it is also important to ascertain if the NP Program has a significant impact across sites and in relation to the current challenges facing access to aged care service provision more generally. For this reason, the quantitative strand of the evaluation approach will assess the delivery models in relation to their cost effectiveness, cost benefit, competitive market contexts and impact on health patterns. To meet these aims, it will be necessary to link NP, client and consultation data collected at delivery model sites with quantitative data kept by the Department of Health and Ageing. The methodology for this data linkage is derived from previous studies conducted by the Australian Institute of Health and Welfare. This linkage of data will occur in five ways. First, the NP Evaluation will seek information about the usage of MBS and PBS item codes according to each individual NP. Access by NPs to MBS and PBS codes is an important policy change in Australia, which intends to increase the financial viability of NP delivery models by offsetting the cost to clients and stabilising the income source of the NP. By knowing the patterns of service use for all patients seen by participating NPs through the study period, the Evaluation will have an indication of the overall extent that this provision is being accessed, the overall level of benefit being paid and the potential economic viability of delivery models. Further, the linkage of NP consultation data to national health administration databases which track health service use in Australia, will allow a historical analysis of health service use by service user prior to and after contact with NP services. Second, the NP Evaluation will seek information about the characteristics of aged care clients using NP service delivery models. This will show the profile of the NP clients. Access to this data is important to assess not only the reach of delivery models, but also their targeting of intense need. Third, the NP Evaluation will seek information about the competitive pressures on NP delivery models. For this reason, the MBS bulk-billing rates for GP medical services for the region around the NP site will be required to assess the competitive environment faced by the NPs. Bulk-billing rates indicate the

59

number of services which service users receive free of charge (due to a publicly funded rebate for the service) and without an additional GP service charge. These rates are higher in regions of low income or greater welfare support and equate to a measure of competitiveness because higher rates indicate lower profit margins for GPs. Further, with the capacity of NPs to provide free bulkbilled services being an important aspect of their market attractiveness compared to GP services, more GPs who bulkbill represents a lower competitive advantage for NPs. Fourth, the NP Evaluation will seek to make assessment of the impact of the introduction of NPs. These comparisons will include temporal standardisation (i.e., before/after test) and contextual standardisation (i.e., intra-site comparison). While it is beyond the scope of the NP Program, funding arrangements and ethics approval to replicate the full evaluation approach in non-NP sites as a control, the evaluation will use national health administration databases to produce a quasi-control measure against which to assess the changing access to aged care services within models. For models in residential aged care facilities, similar facilities without a NP will be identified and their patterns of health service use compared. Further, historical data will be used to compare patterns of health service use in the two years before and after the introduction of a NP service in the same site. These tests aim to respectively minimise the contextual variables within a site and reduce the impact of ongoing change across sites. Finally, the NP Evaluation will also seek information about the specific nature, effectiveness and impact of individual NP service delivery models. This will assist in the assessment of the extent to which particular models are meeting NP Program Goals, as well as reinforce measurements of cost-effectiveness and financial viability. All of this data is to be collated by client characteristics and statistical analysis will provide a detailed perspective of the impact of specific NP models/sites on access to health care by clients. 4.2.3. The case study strand Amongst the objectives of the NP Evaluation approach is to provide ‘real-life’ examples in the complex contexts in which they operate. For that reason, the evaluation approach has incorporated a specific case study strand. This component will serve a number of purposes including providing identifiable examples (which others wishing to establish new NP models might follow), providing heuristics for professional reflection, and providing detailed examinations of the implications of national policies in local contexts. To facilitate this strand, the NP Evaluation approach will adopt the Yin (1994, 2009) model of case study design. Yin proposes a model where case study data is strategically collected to test pre-identified theoretical propositions. The development of propositions in this evaluation approach is based on three sources of information. First, a macro-level review of academic literature about the potential social, economic and population influences on aged care models of practice is conducted. This enables a consideration of drivers that impact on the emergence, competitiveness, economic viability and sustainability of models. Second, a meso-level review of relevant policy literature is conducted to identify key policy and professional drivers that might influence the development, implementation and sustainability of models. Third, a micro-level review of the tender applications for each of the successful models is conducted to identify local level context and unique drivers. All the above drivers are then tested in a first case study. To achieve this, interview, audit, and ethnographic data collection tools provide multi-faceted perspectives on the various models, which are then analysed in relation to the pre-identified propositions. After a case study analysis has been completed, it is used to refine

60

B. Prosser et al. / Evaluation and Program Planning 40 (2013) 55–63

the proposition and then another case study is conducted to retest that proposition. This commences a cyclical approach to data collection with the purpose of producing rigorous analytical generalisation. The main advantage of Yin’s case study design model is that it aggregates around case study examples that are both relevant and accessible to policy-makers. The approach also looks beyond the representativeness of a program to consider what macro level factors are at work within and around the model. Importantly, this component of the NP Evaluation does not seek to present a ‘single view’ of delivery models through case studies; rather it seeks to contribute to an integrated Realistic Evaluation. 4.3. The development process The conceptual frame described above reflects a commitment to reducing the gap between research and policy in ‘real-world’ evaluations. In the view of the authors, this cannot be achieved as an ‘add-on’ where research results are manufactured upstream and left to be translated into evidence by policy-makers downstream (Caplan, 1979). Rather, it is a commitment that is necessary at each stage of evaluation research development from conception to implementation and to reporting (Martin et al., 2011). The first stage of the development process was to involve policy-makers in the design of the conceptual framework and research methodology. Once the Australian Government had selected a successful tender to undertake the NP Evaluation, representatives of the Commonwealth Health and Ageing were involved in consultation around the components and implementation of the evaluation approach. This culminated in the production of an agreed implementation plan, which subsequently guided the progress of the NP Evaluation and was a key topic of discussion in monthly meetings of the Project Steering Committee. Significant outcomes of this part of the development approach were a strengthening emphasis on the use of mixed methods and collection of the views of multiple stakeholders. The second stage of the development process involved collaboration around the content, format and implementation of the different research strands. In November 2011, representatives of the Commonwealth Health and Ageing, along with model managers and NPs from the selected models attended a workshop held in Canberra. At this workshop, all aspects of the evaluation were reviewed and changes made in negotiation with those in attendance. This was followed by the establishment of an online community for those who had attended the workshop so that they could continue to make suggestions up until final ethics approval was granted in July 2012. Significant outcomes of these activities was not only the refinement of data collection tools to be used by NPs to make them more practically useful and accessible, but also an opportunity to negotiate the priorities of the evaluation from both policy-maker and policy practitioner perspectives. The third aspect of the development process was incorporating reference to priorities in policy literature. It was here that Yin’s (2009) case study design approach was most useful. A review of national and international policy literature related to workforce shortages and NP services in aged care was conducted and key themes were identified. These themes were then taken by the evaluation team and adapted into propositions to guide the case study strand. After the first year of site visits and case study analysis, the initial propositions were revisited in light of data collected and through a series of focus group discussions with NPs in a workshop in December 2012. These insights were then used to refine the propositions for use in data collection in 2013.

The fourth aspect of the NP Evaluation process is ongoing, namely, the formative reporting process. This occurs through three mechanisms. First, the monthly Steering Group Meeting discusses the progress of the NP Evaluation against emerging key issues and themes. Since representative of the Commonwealth Department of Health and Ageing are members of this group and also hold responsibility for policy work in this space, this is an important means to provide constant feedback into policy. Second, an annual meeting is convened with senior representatives of the department to discuss not only issues and themes emerging from the data, but also potential policy implications. This meeting is an important means to contribute evidence directly into the policy development cycles of the Commonwealth. Third, a formative report will be produced and provided to the department approximately six months before the final NP Evaluation report is released. It is also proposed that a conference will be convened in 2014 to review the findings with policy-makers, representatives of the models, and the aged care sector. It is important to note that the purpose of the NP Evaluation is to assess a current policy program; hence its funding period encourages formative reporting so that new policy can be developed in preparation for the cessation of the NP Program. 5. Discussion The above conceptual framework includes as part of its development process four important approaches from public health literature that support a stronger research-policy nexus (i.e., including priorities from policy literature, communication and collaboration with multiple stakeholders, accessibility of formative findings for policy-makers, and mixed methods including local cases). It was developed not only as an attempt to evaluate the NP program, but also to address the mutual requirements of rigorous research and evidence-based policy (Petticrew et al., 2004). This is not always an easy task, as there was a time not long ago when the health policy-research gap was seen as a bridge too far, with policy makers and practitioners criticised for being respectively too willing to see research as irrelevant or too slow to adopt research findings into practice (Haines & Jones, 1994). More recently, the growing demand for evidence-based policy has brought the research-policy relationship in public health under renewed scrutiny (Botterill & Hindmoor, 2012; Moat & Lavis, 2012). This scrutiny has more commonly focussed on how research evidence can better inform the practice of practitioners (Gautam, 2008; Henderson, Davies, & Willet, 2006; Martin et al., 2011; Petticrew et al., 2004). However, there has also been recognition of mutual frustration between academic researchers and health policy-makers due to an apparent mismatch between respective research priorities, expectations and epistemologies (Martens, 2005). However, as Martin, Currie, and Lockett (2011) note, the view that researchers and policy-makers lack the will (or ability) to communicate and that the role of the researcher is to bridge the gap (from provider to policy-makers) is now subject to review. Today, researchers and policy-makers, both within and outside of public health, are actively seeking a stronger nexus in their work. For instance, some have argued for a greater focus on systemic reviews of research to inform public health policy-making (Lavis, Becerra-Posada, Haines, & Osei, 2004). Meanwhile, others have argued for new relationships between research, policy and practice that move away from linear transfer of knowledge and toward more collaborative arrangements (Martin et al., 2011), although the feasibility of such approaches is still open to question in current academic and policy context (Martin, Currie, et al., 2012). What this highlights is that the size of the challenge for health policy evaluations is still great. This is borne out by academics

B. Prosser et al. / Evaluation and Program Planning 40 (2013) 55–63

bemoaning a perceived oversimplification of their work, a reticence for independent critique and the constant pressure to pursue external funding, while health policy-makers often complain that they cannot find research reports written in plain language with clear application, which are attuned to current policy settings and produced in time. A recent study of eighty Australian health researchers and policy-makers (Campbell et al., 2009) found that policy makers rarely used research to shape policy agendas and struggled to obtain useful research syntheses; a finding borne out by the above policy history of the NP role in Australia. This study also found that only one third of academic researchers had developed targeted strategies to inform health policy-makers of their research findings, despite there being a shared emphasis on the importance of mutual perspectives from policy and the academy. In response, Campbell and colleagues recommended five general principles to increase the use of research within health policy. First, research findings must be translated into forms that are accessible, relevant, real and ready-to-go for policy-makers. Second, established networks, strong relationships and constant interaction between policy-makers and researchers are vital. Third, deliberate efforts must be made within policy contexts to address historical and structural barriers to research receptivity, while it should be noted that policy-makers pay more attention to research when they have paid for it. Fourth, greater rewards need to be provided within university structures for academics to engage with health policy research and evaluations, as currently there is limited incentive to do so (Martin, Currie, et al., 2011). Finally, the academy must recognise that while real heath policy impact is possible, it will be time consuming and costly (Martens, 2005). Previously, Petticrew et al. (2004) explored similar issues in some detail through a study conducted with seven senior health policy makers in the United Kingdom. From the perspective of these policy-makers, there were serious limitations within the body of public health research. They found that policy-makers see public health research and evaluation as theoretically weak, overlooking issues of equity, lacking a predictive element which is vital for policy, neglecting issues of cost-effectiveness and needing better methods to assess impact on health outcomes. Further, they noted that key issues are often overlooked by health evaluators, such as the need for local case study in addition to national trend data, as well as the time and political pressures on policy development. The participants reported that because public servants frequently develop policy without evidence and in a non-linear fashion, there is no inherent onus on using research (or using it accurately). Hence, if researchers were to have policy impact, they need to be involved in interactive partnerships (Moat & Lavis, 2012), contend with particular institutional arrangements (Lavis, 2006) and be pre-prepared with relevant evidence, in an accessible form, at the right time (Lomas, 2000, 2005; Martins, Currie, et al., 2011). Too often health research evaluations and reports, in the views of the participants, were policy free or politically naı¨ve, with more attention on clinical practice than social determinants and costs. According to Petticrew et al. (2004), what is needed is a greater use of mixed methods and sources to produce plausible evidence-based responses to policy problems. Also identified as a priority, was the development of new conceptual and methodological approaches that could embrace both policy and research priorities. Recently, Haynes et al. (2012) have added another layer of complexity through a consideration of political and media influences. Their research with Australian public servants, ministerial advisers and ministers reiterated the importance of relationships and trust between health researchers and policymakers, as has been championed by Lomas (2000, 2005). Haynes

61

et al. (2012) found that in the majority of cases the most effective method to influence policy was through strong relationships with public servants, although these can often be hard to form and maintain in the context of high staff turnover and the complex bureaucracy within departments. As a result, what is recommended is that public health researchers develop a multifaceted approach to increasing policy influence. In light of this, it is insightful to briefly note the multi-faceted United Kingdom Collaborations for Leadership in Applied Health and Research and Care (CLAHRCs) as described by Martin et al. (2011). These nine projects aim to carry out health research, implement findings in local healthcare organisations and build capacity for using evidence in health practice. In some ways, the NP Evaluation can be seen as similar to these CLAHRCs through their mutual pursuit of generative dialogue through between multiple stakeholders, shift beyond linear evaluation models that focus solely on program objectives, and commitment to formative communication between researchers and policy-makers. The CLAHRCs also share an emphasis on mixed methods and a regional focus and in some cases the use of Realistic Evaluation. However, there is an important difference in that the NP Evaluation is more orientated toward matters of accessibility, economic viability and the success of a policy program, while the CLAHRCs models have a far greater emphasis on evidence informing better clinical practice and greater support for assessing policy impact. That said, there are many shared challenges around the evaluation diverse activities, theoretical rigour, selecting appropriate methods, negotiating the often conflicting views of stakeholders and assessing sustainability. It is such challenges that the NP Evaluation has also sought to address as part of its design. 6. Conclusion This is an overview of an evaluation approach currently being used to evaluate the impact of aged care NP services in Australia. While located in this specific context, this paper illustrates a response to emerging challenges for research and policy that are similarly experienced in other western nations. To do so, the paper has provided a review of the policy history behind the establishment of the NP role in Australia, as well as an overview of the aims that underpin both an initiative to support innovative models of Nurse Practitioner aged care practice, and an independent evaluation of that initiative. Significantly, this approach strives to extend beyond a model of policy evaluation that is purely descriptive or aims for the assessment of model outcomes to policy program aims. Hence, the paper describes the development of an evaluation approach that seeks to be accessible, realistic and rigorous for researchers and policy-makers alike, with a view to providing sufficient detail for the approach to be considered for use by others evaluating public health policy programs in the future. Acknowledgements This paper is a contract material developed within the ‘Nurse Practitioner Aged Care Models of Practice National Evaluation’ project, which has been funded by the Australian Government Department of Health and Ageing. The authors also wish to acknowledge the contribution of the following University of Canberra and Australian National University researchers to the conceptual development and implementation of this project: Professor Rachel Davey (project director); Dr. Shannon Clark; Professor Diane Gibson; Adjunct Assoc. Professor John Goss; Dr. Catherine Hungerford; Dr. Carmel McQuellin; Assoc. Professor Rhian Parker; Dr. Brenton Prosser.

62

B. Prosser et al. / Evaluation and Program Planning 40 (2013) 55–63

Lessons Learned As identified in this paper, the NP Evaluation approach shared a number of similarities with the multi-site CLAHRCs initiative, most notably, the value of integrating mixed methods into the framework for evaluations of complex, wide-ranging and multi-site programs (which has also been discussed extensively in previous editions of this journal). Amongst the often cited advantages for evaluators adopting such approaches is the capacity to consider both different stakeholder priorities and diverse approaches to meeting program goals. In the case of the NP framework, the use of theoretically driven propositions to guide case study analysis can reduce the risk of over-emphasis on program processes at the expense of rigorous analysis. Gathering quantitative data on patterns of service use and linking these to national health databases allows for generalisation across models, can support assessments of health impact and can lead to identification of social patterns. The inclusion of considerations of cost-effectiveness and financial viability can directly addresses criticism from policy-makers that too often this information is omitted from public health evaluations. However, incorporating qualitative analysis of participant responses can also capture the diversity of different models and uncover the meanings that different actors attribute to service delivery approaches, as well as identify insights that might be excluded within more deductive approaches. This process of mapping the diversity of projects, situations, actors and points of view also benefits from a combination of methods. Hence, the combination of different types of data provides a better understanding of the potential impacts of complex programs for and with multiple stakeholders. While each data collection method within a mixed method approach has its own limitations, the use of several data sources within this framework to support triangulation can help overcome these limitations by providing rich and multiple perspectives on the same phenomena. Further, when used independently (according to rigorous methodological standards), findings from different methodological approaches that reinforce each other can be used to identify a higher order of validity. Given the greater utility of research syntheses and the demand for reliable results amongst policy-makers, such an approach presents obvious advantages in bridging the often cited gap between researcher and policy-maker knowledge requirements. This makes it possible to not only report reliably on a program’s impact, but also shed light on specificities in their social contexts. Mixing methods in a rigorous framework makes it easier to develop an unfolding understanding of a program’s short and long term impacts, while building in the regular report of formative findings can assist in ongoing policy program development. The conditions for formative feedback are an important aspect of frameworks that seek to reduce the gap between research and policy in ‘real-world’ evaluations. Drawing on a common theme within the literature in this area, this approach has actively involved a range of stakeholders (including policymakers and practitioners), in consultation about the conceptual and methodological aspects of the developing framework. Evaluators should participate in such consultation early and often, and this framework has within it a number of examples of

how this might be carried out through the life of a project. The advantages of ongoing communication extend beyond practical guidance during implementation and management of stakeholder expectations; it can also help streamline complex administrative requirements when seeking to access national health databases. Another advantage of ongoing collaboration with stakeholders is that it provides opportunity to refine different forms of reporting to the needs and interests of the broad range of audiences involved with complex policy programs. The NP Evaluation also shared a number of practical challenges with the CLAHRCs study, which are worthy of consideration by evaluators. First, the need to negotiate governance and ethics requirements across thirty-two sites that had been selected based on their diversity (in many cases with no ethics precedent in place) was a time-consuming and labour intensive task. Second, the involvement of NPs in data collection activities in addition to their daily responsibilities was time also consuming and strategies to reduce this burden on professional colleagues need to be included in evaluator planning. Third, while the NP Evaluation has developed a framework that seeks to increase its potential for policy influence (as it considers both NP Program aims and NP Evaluation objectives), it is important to note that collection of empirical data on its direct policy influence (as has commenced with the CLAHRCs study) is beyond the scope and resources of the NP Evaluation. In summary, this paper has described an evaluation framework that also aims to support accessibility for policy-makers. This is achieved through its deliberate inclusion of policy propositions, its collaborative consultation strategy, its focus on local case studies, and its specific consideration of health impact. In doing so, it demonstrates the pursuit of theoretical rigour while maintaining a practical orientation, with the result being an innovative methodological approach to evaluating complex initiatives in real life contexts.

References Agosta, L. J. (2009). Patient satisfaction with nurse practitioner-delivered primary healthcare services. Journal of the American Academy of Nurse Practitioners, 21, 610–617. Auerbach, D. I. (2012). Will the NP workforce grow in the future? New forecasts and implications for healthcare delivery. Medical Care50. Australian Government. (2010). National Health (Collaborative arrangements for nurse practitioners) determination 2010 URL (Accessed 11 October 2012): http:// www.comlaw.gov.au/Details/F2010L02107. Australian Nursing and Midwifery Council. (2012). National Competency Standards for Nurse Practitioners URL (Accessed 11 October 2012): http://www.nursingmidwiferyboard.gov.au/documents/default.aspx?record=WD10%2f1351&dbid=AP&chksum=MhZbIQy5eHKHdz0u%2bhCSTg%3d%3d. Baum, F. (2009). Community health services in Australia. In J. Germov (Ed.), Second opinion: An introduction to health sociology (pp. 499–521). Oxford: Oxford University Press. Botterill, L. C., & Hindmoor, A. (2012). Turtles all the way down: Bounded rationality in an evidence-based age. Policy Studies, 33, 367–379. Brown, S. A., & Grimes, D. E. (1995). A meta-analysis of nurse practitioners and nurse midwives in primary care. Nursing Research, 44, 332–339. Campbell, D. M., Redman, S., Jorm, L., Cooke, M., Zwi, A. B., & Rychetnik, L. (2009). Increasing the use of evidence in health policy: Practice and views of policy makers and researchers. Australia and New Zealand Health Policy, 6, 21–32. Caplan, N. (1979). The two-communities theory and knowledge utilization. American Behavioral Scientist, 22, 459–470. Christian, R., & Baker, K. (2009). Effectiveness of nurse practitioners in nursing homes: A systematic review. JBI Library of Systematic Reviews, 7, 1333–1352. Davidson, P., Halcomb, E., Hickman, L., Phillips, J., & Graham, B. (2006). Beyond the rhetoric: what do we mean by a ‘model of care’? Australian Journal of Advanced Nursing, 23, 47–55.

B. Prosser et al. / Evaluation and Program Planning 40 (2013) 55–63 Dick, K., & Frazier, S. C. (2006). An exploration of nurse practitioner care to homebound frail elders. Journal of the American Academy of Nurse Practitioners, 18, 325–334. Dow, B., & McDonald, J. (2007). The invisible contract: Shifting care from the hospital to the home. Australian Health Review, 31, 193–202. Department of Health and Ageing. (2011). New nurse practitioners to care for older Australians URL (Accessed 11 October 2012): http://www.health.gov.au/internet/ ministers/publishing.nsf/Content/mr-yr11-mb-mb225.htm. Donald, F., Martin-Misener, R., Bryant-Lukosius, D., Kilpatrick, K., Kaasalainen, S., Carter, N., et al. (2010). The primary healthcare nurse practitioner role in Canada. Nursing Leadership, 23, 88–113. Duckett, S. (2007). The Australian health care system (3rd ed.). South Melbourne: Oxford University Press. Gautam, K. (2008). Addressing the research-practice gap in healthcare management. Journal of Public Health Management & Practice, 14, 155–159. Glasgow, R. R., Klesges, L. M., Dzewaltowksi, D. A., Estabrooks, P. A., & Vogt, T. M. (2006). Evaluating the impact of health promotion programs: Using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Education Research, 21, 688–694. Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 39, 1322–1327. Haines, A., & Jones, R. (1994). Implementing research findings. British Medical Journal, 38, 1488–1492. Haynes, A. S., Derrick, G. E., Redman, S., Hall, W. D., Gillespie, J. A., Chapman, S., et al. (2012). Identifying trustworthy experts: How do policymakers find and assess public health researchers worth consulting or collaborating with? PLoS ONE, 7, e32665 http://dx.doi.org/10.1371/journal.pone.0032665. Henderson, A., Davies, J., & Willet, M. (2006). The experience of Australian project leaders in encouraging practitioners to adopt research evidence in their clinical practice. Australian Health Review, 30, 474–484. Horrocks, S., Anderson, E., & Salisbury, C. (2002). Systematic review of whether nurse practitioners working in primary care can provide equivalent care to doctors. British Medical Journal, 324, 819–823. Jolly, R. (2009). Bills digest: Health Legislation Amendment (Midwives and Nurse Practitioners) Bill 2009. Parliamentary Library of Australia URL (Accessed 11 October 2012) http://www.aph.gov.au/binaries/library/pubs/bd/2009-10/10bd011.pdf. Laurant, M. G., Hermens, R. P., Braspenning, J. C., Akkermans, R. P., Sibbald, B., & Grol, R. P. (2008). An overview of patients’ preference for, and satisfaction with, care provided by general practitioners and nurse practitioners. Journal of Clinical Nursing, 17, 2690–2698. Laurant, M. G., Hermens, R. P., Braspenning, J. C., Sibbald, B., & Grol, R. P. (2004). Impact of nurse practitioners on workload of general practitioners: Randomised controlled trial. British Medical Journal, 328, 927. Lavis, J. N. (2006). Research, public policymaking, and knowledge-translation processes: Canadian efforts to build bridges. Journal of Continuing Education in the Health Professions, 26, 37–45. Lavis, J. N., Becerra-Posada, F., Haines, A., & Osei, E. (2004). Use of research to inform public policymaking. The Lancet, 364, 1615–1621. Lomas, J. (2000). Using ‘linkage and exchange’ to move research into policy at a Canadian foundation. Health Affairs (Millwood), 19, 236–240. Lomas, J. (2005). Using research to inform healthcare managers’ and policy makers’ questions: From summative to interpretive synthesis. Healthcare Policy, 1, 1. Martens, P. J. (2005). When health services researchers and policy makers interact: Tales from the teutonic plates. Health Care Policy, 1, 72–84. Martin, G. P., Currie, G., & Lockett, A. (2011). Prospect for knowledge exchange in health policy and management: Institutional and epistemic boundaries. Journal of Health Service Research & Policy, 16, 211–217. Martin, G. P., Ward, V., Hendy, J., Rowley, E., Nancarrow, S., Heaton, J., et al. (2011). The challenges of evaluating large-scale, multi-partner programmes: The case of NIHR CLAHRCs. Evidence & Policy: A Journal of Research, Debate and Practice, 7, 489–509. McCabe, M. S., & Jacobs, L. A. (2012). Clinical update: survivorship care-models and programs. Seminars in Oncology Nursing, 28, e1–e8.

63

McKenzie, R., Naccarella, L., Stewart, A., & Thompson, C. (2007). Targeting what matters in health promotion evaluation: Using the RE-AIM approach to identify success in real-world settings. Evaluation Journal of Australasia, 7, 19–46. Meyer, M., Davis, M., & Mays, G. (2012). Defining organizational capacity for public health services and systems research. Journal of Public Health Management and Practice, 18, 535–544. Moat, K. A., & Lavis, J. N. (2012). 10 best resources for. . . evidence-informed health policy making. Health Policy and Planning http://dx.doi.org/10.1093/heapol/ czs050. Mundinger, M. O., Kane, R. L., Lenz, E. R., Totten, A. M., Tsai, W., & Cleary, P. D. (2000). Primary care outcomes in patients treated by nurse practitioners or physicians: A randomized trial. Journal of the American Medical Association, 283, 59–68. National Health Workforce Taskforce. (2009). Health workforce in Australia and factors for current shortages KPMG. URL (Accessed 12 October 2012): http://www.ahwo.gov.au/documents/NHWT/The%20health%20workforce%20in%20Australia%20 and%20factors%20influencing%20current%20shortages.pdf. Nursing and Midwifery Board of Australia. (2012). Registered nurse and enrolled nurse data March 2012. URL (Accessed 10 July 2012): http://www.nursingmidwiferyboard.gov.au/documents/default.aspx?record=WD12%2f8117&dbid=AP&chksum=NSb%2bWfse5gDR6Cv6KArupw%3d%3d. Parker, R., Forrest, L., Ward, N., McCracken, J., Cox, D., & Derrett, J. (2013). How acceptable are primary health care nurse practitioners to Australian consumers? Collegian, 20, 35–41. Pawson, R., & Tilley, N. (1997). Realistic evaluation. Thousand Oaks: Sage. Pawson, R., & Tilley, N. (2004). Realist Evaluation URL (Accessed 11 October 2011): http://www.communitymatters.com.au/RE_chapter.pdf. Petticrew, M., Whitehead, M., Macintyre, S. J., Graham, H., & Egan, M. (2004). Evidence for public health policy on inequalities: 1: The reality according to policymakers. Journal of Epidemiology and Community Health, 58, 811–816. Productivity Commission. (2005). Economic implications for an ageing Australia URL (Accessed 11 October 2012): http://www.pc.gov.au/projects/study/ageing/docs/ finalreport. Rycroft-Malone, J., Fontenla, M., Bick, D., & Seers, K. (2010). A realistic evaluation: The case of protocol-based care. Implementation Science, 5, 1–14. Sibbald, B., Laurant, M. G., & Reeves, D. (2006). Advanced nurse roles in UK primary care. Medical Journal of Australia, 185, 10–12. Smyth, R. (2004). Exploring the usefulness of a conceptual framework as a research tool: A researcher’s reflections. Issues in Educational Research, 14, 167–180. Yin, R. (1994). Case study research (2nd ed.). Thousand Oaks: Sage. Yin, R. (2009). Case study research (4th ed.). Thousand Oaks: Sage. Brenton Prosser is a senior research fellow to the Nurse Practitioners Aged Care Models of Practice Evaluation within the Centre for Research and Action in Public Health, University of Canberra. He is an adjunct in the School of Sociology at the Australian National University and in the School of Education at the University of South Australia. Shannon Clark is a post-doctoral fellow in the Centre for Research and Action in Public Health, University of Canberra. She is also a visiting fellow at the Australian Primary Health Care Research Institute at the Australian National University. Rachel Davey is Director of the Centre for Research and Action in Public Health, University of Canberra. Her research focusses on the primary and secondary prevention of chronic disease and in evaluating public health interventions and innovative models of health service delivery. Rhian Parker is a member of the Australian Primary Health Care Institute, Australian National University. Her research focusses on the organisation and delivery of primary health care services, with particular focus on nurses and preventive activities and on knowledge translation.