- Email: [email protected]
Diversity training for the community aged care workers: A conceptual framework for evaluation
Accepted Manuscript Title: Diversity training for the community aged care workers: A conceptual framework for evaluation Authors: Arti Appannah, Claudia Meyer, Rajna Ogrin, Sally McMillan, Elizabeth Barrett, Colette Browning PII: DOI: Reference:
S0149-7189(16)30192-6 http://dx.doi.org/doi:10.1016/j.evalprogplan.2017.03.007 EPP 1432
To appear in: Received date: Revised date: Accepted date:
5-9-2016 12-3-2017 21-3-2017
Please cite this article as: Appannah, Arti., Meyer, Claudia., Ogrin, Rajna., McMillan, Sally., Barrett, Elizabeth., & Browning, Colette., Diversity training for the community aged care workers: A conceptual framework for evaluation.Evaluation and Program Planning http://dx.doi.org/10.1016/j.evalprogplan.2017.03.007 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Title Diversity training for the community aged care workers: A conceptual framework for evaluation Funding This work was supported by the Department of Social Services, Australia. Authors Ms Arti APPANNAH (corresponding author)1 BachSocScience (Hons), MPsych (I/0), PhD Candidate 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
Email: [email protected] Dr Claudia Meyer 1 BachAppScience (Physio), MPubHealth, PhD 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
2
La Trobe University, Centre for Health Communication and Participation, Plenty Road & Kingsbury Drive, Bundoora, Victoria, 3086, Australia Email: [email protected] Phone: 61 3 8530 8105 Dr Rajna Ogrin1 BSc, BPod (Hons), PhD 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
2
University of Western Ontario, 151 Richmond St, Canada
3
University of Melbourne, Austin Health Department of Medicine, 145 Studley Rd, Heidelberg, Victoria, Australia Email: [email protected] Ms Sally McMillan1 GradCert (Anaphylaxis Management Training), Dip VocEducTrain, Registered Nurse 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
Email: [email protected] Ms Elizabeth Barrett1 GradCertCareerCoun, DipProjMan, DipPosPsych, GradDipHumanResources, MBusLeader 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
Email: [email protected] Prof Colette Browning1 BSc (Hons), MSc, PhD 1
1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
2
Monash University, Wellington Rd & Blackburn Rd, Clayton, Victoria, Australia
3
International Institute for Primary Health Care Research, Shenzhen, China
Email: [email protected] Highlights
What this paper adds? As little is known about the effectiveness of diversity training in the community aged care industry, the construction of an evaluation framework provides an empirical and consistent method to assess the impact of this kind of training An existing training and evaluation framework is used to synthesise findings from the literature and construct a formative evaluation framework specific to diversity training programs in the community health and aged care industry The evaluation framework integrates three learning outcomes; cognitive, affective and skilledbased learning outcomes and presents a mixed methods approach for evaluating diversity training programs in the community aged care industry Abstract
Older Australians are an increasingly diverse population, with variable characteristics such as culture, sexual orientation, socioeconomic status, and physical capabilities potentially influencing their participation in healthcare. In response, community aged care workers may need to increase skills and uptake of knowledge into practice regarding diversity through appropriate training interventions. Diversity training (DT) programs have traditionally existed in the realm of business, with little research attention devoted to scientifically evaluating the outcomes of training directed at community aged care workers. A DT workshop has been developed for community aged care workers, and this paper focuses on the construction of a formative evaluative framework for the workshop. Key evaluation concepts and measures relating to DT have been identified in the literature and integrated into the framework, focusing on five categories: Training needs analysis; Reactions; Learning outcomes, Behavioural outcomes and Results. The use of a mixed methods approach in the framework provides an additional strength, by evaluating long-term behavioural change and improvements in service delivery. As little is known about the effectiveness of DT programs for community aged care workers, the proposed framework will provide an empirical and consistent method of evaluation, to assess their impact on enhancing older people’s experience of healthcare.
Keywords: Diversity training; Community care; Evaluation
2
Introduction By 2050, over 3.5 million older people in Australia will access community aged care services annually, with 80% of these services delivered in the community (ACSA, 2015). As Australians live longer, there will be increasing demands for aged care services to deliver more complex care in the community, and for an aged care workforce that can meet the needs of increasingly older populations from diverse backgrounds and with diverse needs. To ensure care provision is appropriate and leads to optimal health outcomes, the many characteristics that define an individual need to be considered in care assessment and planning. Further, there is growing acknowledgement that some populations are at risk of marginalisation and will require additional support within the aged care system to ensure equitable access and care; e.g. Aboriginal and Torres Strait Islanders; people from Culturally and Linguistically Diverse (CALD) backgrounds; people with dementia; people living in rural and remote areas; people experiencing financial disadvantage; people who are/or at risk of homelessness; and people who are lesbian, gay, bisexual, transgender and intersex (LGBTI) (Australian Institute of Health and Welfare, 2015). Therefore, training to raise awareness of diverse characteristics and their impact on access to healthcare becomes increasingly important for aged and healthcare workers. The need for delivering diversity training (DT) to age and healthcare workers has gained attention from policy makers, aged care services, professionals and educators as a strategy to improve outcomes in aged care; enhancing client participation in these services through diversity appropriate care (ACSA, 2015; Horvat, Horey, Romios, & Kis-Rigo, 2014). Diversity appropriate care is care in which professionals pay deliberate attention to the specific needs of an individual following from the interplay between characteristics that make up diversity in an individual, including but not limited to the broad groupings of gender, ethnic origin and socio-economic status (Celik, Abma, Klinge, & Widdershoven, 2012). Research on health disparities suggests that people who receive culturally sensitive care are likely to show an increased level of adherence to medical advice and to report greater satisfaction with their care providers (Chenowethm, Jeon, Goff, & Burke, 2006; Cook, Kosoko-Lasaki, & O'Brien, 2005). To extrapolate from this, it is possible that awareness of diversity appropriate care can yield similar outcomes to enhance an individuals’ experience of healthcare. An older person may have a number of diverse characteristics that may or may not impact upon their participation in healthcare. For example: an older person with low health literacy, who lives in a remote area, belongs to the LGBTI community and has multiple morbidities may have differing health needs from an older person with good health literacy, who lives in a metropolitan region and has similar multiple morbidities. Differing characteristics of a person can impact on appropriate access and delivery of health services. A term coined ‘intersectionality’ has been premised in the diversity literature; that humans are made up of more than a single characteristic, and that these characteristics are prioritised by individuals in different ways, and should not be viewed through siloed lenses, and, importantly, that their interplay requires special consideration (Bagilhole, 2010; Hankivsky et al., 2014). Healthcare professionals are required to recognise diverse characteristics in their clients, yet DT interventions to increase awareness of diversity appropriate care in the aged care industry are few. Awareness about diversity appropriate care in healthcare practices can be stimulated by providing health professionals with diversity awareness training (Celik et al., 2012), yet, to date this training has mainly been in the form of cultural competence training (Horvat et al., 2014). Other types of training have started to emerge in the literature such as Lesbian, Gay, Bisexual, Transgender and/or Intersex (LGBTI) training (Spina, 2015) or dementia care training for aged care workers (Pleasant et al., 2016). Yet these types of training again focus on one diversity characteristic through a siloed 3
lens, not discussing ‘diversity within diversity’ where people are made up of multiple diversity characteristics, some of which may have positive impacts on access to healthcare. Consequently, little research attention has been devoted to scientifically and systematically evaluating the effectiveness of DT programs (Pendry, Driscoll, & Field, 2007), particularly in the health and aged care industry. Indeed, Cocchiara, Connerley, and Bell (2010) acknowledge the inconsistent use (or non-use) of specific measurement criteria when evaluating DT, hence leading to difficulties in gauging the effectiveness of these types of interventions. Studies that have evaluated the outcomes of DT programs have predominantly used self-report measures collected from a single source, without assessing the long-term effects of learning outcomes (Bezrukova, Jehn, & Spell, 2012; Kulik & Roberson, 2008). Few DT programs for community health and aged care workers are documented in the literature, and thus little is known about the effectiveness of these programs. The evaluation of DT interventions is essential to measure their impact on the knowledge, skills and service delivery of health and aged care workers, as well as outcomes on clients (i.e. whether the training leads to healthcare that meets the needs of the client). A consistent and systematic approach to evaluating the outcomes of DT workshops can help inform their design and delivery, providing more effective services to older people accessing health and aged care and ultimately enhance their participation in health services. A DT program for community health and aged care workers has been developed and is underpinned by human rights, psychological and sociological theories (Meyer et al., 2016). The training assists participants to identify a broad range of diversity characteristics for older people in health and aged care, examines perspectives and assumptions related to diversity, and determines initiatives to improve service provision through the use of an action plan. The training is novel in community health and aged care since its focus is not solely on culture, and it moves away from a siloed view of diversity, addressing the ‘intersectionality’ of diversity characteristics in a person (Bagilhole, 2010; Hankivsky et al., 2014). The purpose of this paper is to draw from both the DT and evaluation literature to present an evaluation framework that is specific to DT for health and aged care workers and to comprehensively evaluate the developed training This paper reports on a narrative review of the literature, with findings synthesised according to revised categories in the Kirkpatrick model of training and evaluation (J. Kirkpatrick & Kirkpatrick, 2009). A framework specific to evaluating DT programs for community health and aged care workers is proposed, providing a consistent and empirically sound method of evaluation, to ultimately enhance training delivery and the provision of diversity appropriate care. Methodology for Literature Review and Synthesis A review and synthesis of the literature relevant to diversity programs and evaluation was conducted using business, sociology and health related electronic databases (SAGE Management and Organisational Studies; CINAHL; Informit Business; EMBASE; MEDLINE; Proquest Nursing and Allied Health; Proquest Psychology; ERIC; Proquest Health Management; Proquest Sociology; Business Source Complete; PsychINFO; and Proquest Business) and Google Scholar. The search of titles and abstracts was conducted from 1900s to July 2015, published in English, in order to identify the relevant training and evaluation literature, theories and models. The following search terms were used relevant to diversity: diversity; diversity theory; diversity model; diversity framework; and diversity training. The following search terms were used relevant to evaluation: evaluation; training evaluation; evaluation framework; evaluation methodologies and diversity training evaluation. All types of studies were included; randomised controlled trials, uncontrolled evaluations, qualitative research and descriptive pieces. 4
From 4599 articles screened by reading the title and abstracts of the papers, 28 papers relevant to DT and evaluation were found. The literature was reviewed and key concepts were identified relevant to the evaluation of DT. The concepts were then summarised into an all-inclusive, standardised framework for conducting an evaluation of a DT program within the aged care industry. A revised model of training and evaluation (Kirkpatrick and Kirkpatrick, 2009) depicted in Figure 1 was used to synthesise the findings of the review. This model of training and evaluation was based on an original model developed by Kirkpatrick (1959). The stages of the identified evaluation steps from the literature are discussed within the context of DT for the aged care industry, with relevant information included on instruments and methods of evaluation. Results Key steps for a formative evaluation framework for the community aged care sector has been identified through a review of the DT evaluation literature, using Kirkpatrick and Kirkpatrick’s (2009) model of training and evaluation. The findings have been categorised according to the following key components of the model: 1. training needs analysis; 2. reaction criteria; 3. learning criteria; 4. behaviour criteria; and 5. results. Each component will be discussed. Insert Figure 1 Needs analysis Kirkpatrick and Kirkpatrick (2009) identified training needs analysis as the first step to delivering and evaluating training, where critical behaviours, required competencies, learning objectives and necessary learning environments are established. According to Brown (2002), this training needs analysis should be an ongoing process where relevant data is gathered to help organisations determine whether their training objectives have been attained through the program. A needs analysis conducted prior to the training program can also provide a baseline assessment of the participants’ knowledge, assisting to direct the content of the workshop appropriately. Arthur Jr, Bennett Jr, Edens, and Bell (2003) purport that a thorough and methodical needs assessment is not only integral to the design and development of the training, but also to its evaluation by specifying a number of key features of the training for measuring the outcomes of the program (such as identifying what aspects of the training can be used to evaluate behavioural or learning outcomes). The training needs analysis is important for evaluation because unless informational needs are developed prior to the training, the measurement of the training effectiveness can be difficult (Brown, 2002). To assist in evaluation, Gould, Kelly, and White (2004) suggest that the outcomes of the training needs analysis should be clearly stated and matched to the original aims, organisational needs and requirements of the stakeholders. Roberson, Kulik, and Pepper (2003) also suggest that within a needs assessment, the strength and direction of trainees’ attitudes toward diversity, exposure to diversity-related issues (e.g. whether previous DT has been received), and potential inconsistencies between diversity-related attitudes and behaviours should be assessed. In undertaking a needs analysis, a three step process has been proposed by McGehee and Thayer (1961) which consists of an organisational analysis about where the training can be used in the organisation, an operations analysis to determine what should be included in the training, and a person analysis to find out about how employees are currently performing in the organisation. Similarly, Goldstein and Ford (2002) state that an organisational analysis is undertaken as a systemwide analysis beyond task and person, to determine the state of environmental conditions and 5
whether there are any barriers to the training. Resource requirements for the training and the availability of resources is needed as part of the organisational analysis (Goldstein & Ford, 2002) A needs analysis should consider the prior knowledge of the participants (Cocchiara et al., 2010), the organisational objectives and the characteristics of the task to be learned (Alvarez, Salas, & Garofano, 2004). Arthur Jr et al. (2003) adapts this process to suit a diversity context, proposing the three steps to be: 1. Organisational analysis: To determine support of diversity training in the organisation and organisational needs; 2. Task analysis: To understand the diversity training needs for employees to perform their job effectively; and 3. Person analysis: To decide on which individuals in the organisation require the training. To capture these tenets of information within a needs analysis, a questionnaire can be developed and sent to the trainees a few months prior to the workshop. According to the three-step process outlined above, the organisational analysis poses questions related to the organisation’s approach to diversity, current levels of organisational support and policy offered for the provision of inclusive services, and current processes in place in the organisation (e.g. diversity care plans and resources , to gauge the organisation’s investment in diversity related activities. For the task analysis, current staff knowledge on diversity, their understanding of diversity as a concept and how it can be applied in their day-to-day role as an aged care worker can be determined through the provision of a case study example of a client with diverse characteristics. The case-study allows employees to identify their perceptions of a client’s diverse characteristics, how these diverse characteristics may impact a client’s participation in healthcare, and what usual care entails for clients with diverse characteristics. Targeted behaviours to change for the training, learning objectives and success indicators (such as successfully developing and implementing an action plan relevant to diversity) can be identified in the task analysis. For the person analysis, the target audience for DT in the organisation is identified (e.g. both managers and frontline workers), including the optimal ratio of managers and frontline workers to include in the program so as to foster organisational support for a diversity approach in aged care. Reaction criteria Reaction criteria can be defined as the degree to which participants react favourably to the training (Kirkpatrick, 1987). Reaction criteria are usually operationalised through self-report, representing trainees’ affective and attitudinal (i.e. participants’ feelings towards the training) responses to the training program. Horvat et al. (2014) devised a four-dimensional conceptual framework for describing and assessing cultural competence in healthcare. While slightly different to DT, it expands on a pedagogical approach that includes the reactions or satisfaction with the teaching and learning methods used and the theoretical constructs that underpin it. Moreover, anecdotal evidence suggests that reaction measures are the most widely used evaluation criteria in applied settings (Arthur Jr et al., 2003). Drawing on the pedagogical approach proposed by Horvat et al. (2014), a survey consisting of items that assess satisfaction with the teaching and learning method, the practical applicability of the training content, whether the objectives of the training were attained, and their general satisfaction and enjoyment of the training delivery (Celik et al., 2012) can be used to evaluate reaction criteria. Example items that can be included in the survey are “I was satisfied with the way this program was delivered”, “I was able to meet the key learning objectives because of the way the training was presented”, and “I enjoyed participating in this workshop”. 6
Specific to DT in health and aged care, satisfaction items can also address whether the training aims and learning objectives were clear, whether the diversity training provided them with relevant and practical information for their jobs, and whether they felt the training on a whole helped them in their jobs. Example items are “the training key learning objectives were clearly defined”, “This diversity training did not provide me with information that was practical and that could be easily used in my daily work”, and “By participating in this training I improved my skills to more effectively meet the needs of our diverse community”. Satisfaction items developed to assess reactions to DT in community health and aged care are in supplementary file A. Independent observers can also assist in gauging reactions to the training and providing critical feedback for subsequent training workshops. For example, observers can make note of the variations in level and quality of discussions in the training, or a lack of engagement with a particular topic (Johnson, 2008). Using independent observers can enhance the validity of self-report measures, by rating the required knowledge and skill development (Kraiger, Ford, & Salas, 1993), the pedagogical methods used, the delivery and format of the workshop (Horvat et al., 2014), reactions to the training (D. L. Kirkpatrick, 1987) and participant engagement. Bezrukova et al. (2012) also state that the use of independent observers provide a more objective measure of the training, particularly in comparison to using solely self-report measures typically used for evaluating reactions. The use of forms and checklists can be used to capture information for the objectives of the evaluation program, assisting in providing formal criteria for the evaluation of training by the independent observer (Phillips, 1997). Types of factors to observe include the: extent to which training objectives appeared to be obtained (Colton & Covert, 2007); factors influencing training objectives attainment (including content, clarity of explanations, instructor’s knowledge of the subject, the use of training aids, appropriateness of the material to the group) (Horvat et al., 2014; Schneier, 1994); presence, absence, frequency or duration of participants’ behaviours as they occur (Brinkerhoff, Brethower, Nowakowski, & Hluchyj, 2012); and interpretive comments on the intentions or attitudes which might underlie the various observed behaviours (Celik et al., 2012). An independent observer rating sheet developed for DT training in community health and aged care can be found in supplementary file B. These guidelines can assist in capturing important information for the evaluation of a training program, while avoiding the biases that accompany more subjective evaluation measures (Groot, 2000). Learning criteria A conceptual basis for assessing learning outcomes from the training was first developed by Kirkpatrick (1976; 1987). He indicates that learning criteria evaluate the degree to which trainees acquire the knowledge, skills and attitudes from the training. There are some flaws with this model, most prominently the lack of clarity regarding what specific changes to learning may occur as a result of the training and what assessment techniques would be appropriate to measure these changes (Alliger & Janak, 1989; Kraiger et al., 1993). To address these concerns, Kraiger et al. (1993) advance Kirkpatrick’s (1987) model by defining the three categories of learning outcomes which help to explain the types of changes expected to occur following the training. These outcomes were specifically: 1. cognitive based (the knowledge gained from the training, including verbal knowledge and strategies); 2. skill-based (the behavioural changes resulting from the training); and 3. affective-based (attitudes, self-efficacy, and motivation in general). Kalinoski et al. (2013) contributed to the affective component of this model, proposing that
7
cognitive and skill-based outcomes may reflect relatively explicit outcomes (more conscious), but the affective-based outcomes better reflect both explicit and implicit (more unconscious) components. The content of the evaluation survey would need to include the following: 1. Cognitive-based domain: Questions that assess the types of knowledge acquired, how this knowledge can be applied and the skills developed through knowledge acquisition (Horvat et al., 2014). 2. The skills-based domain: Measures of behaviour and behavioural intention, together with questions on how healthcare workers adapt their service to suit the diverse needs of their clients, how they interact with their organisation to increase awareness about diversity, and how they share information with their client to increase their participation in diversity appropriate healthcare (Kalinoski et al., 2013). 3. The affective-based outcome domain: To contain items that measure the internal states that drive perception and behaviour. These include items that measure attitudes towards older people of diverse needs, perceptions of having diversity policies in an organisation, and feelings relating to dealing with diversity at work (e.g. enthusiastic, stressful, etc) (Kalinoski et al., 2013). For the evaluation of the three learning outcomes of a DT workshop for community aged care workers, to the best of the authors’ knowledge, there is currently no tool available. A DT evaluation survey has been developed to assess the learning criteria, using a Delphi technique. The Delphi technique, predominantly developed by Dalkey and Helmer in 1963, is a widely accepted method of eliciting and converging information from experts within a certain topic area. The method was devised in order to obtain the most reliable opinion agreement from a group of experts on a particular topic by subjecting them to a series of questionnaires, with controlled feedback on their opinions (Dalkey & Helmer, 1963). In round one, specific information about content areas of the topic is solicited from Delphi participants, which can include project and reference group members. In round two, the responses from round one are used to construct the survey instrument which is circulated to the Delphi panellists for further feedback. In the third round, the Delphi subjects receive the survey items and summarised feedback by the researcher, and are asked to revise their judgements in order to make further clarifications. In the final round, the list of remaining items, their ratings, opinions and items achieving consensus are distributed to the panellists, to provide a final opportunity to revise judgments (Hsu & Sandford, 2007). The developed survey measures the three learning outcomes of cognitive, skills and affective domains and will be validated in a subsequent study. Example items measuring the cognitive domain are, “I have the skills and knowledge to provide the right care to older people of diverse characteristics”, “The diverse characteristics of a client can be used positively to assist in their participation in healthcare”, and “In my work, I believe that I am capable of dealing appropriately with older people of diverse characteristics”. Examples of items measuring the skill-based domain are “When providing healthcare/community aged care services, I always appropriately adapt services in a way that includes the older person”, “I usually take the time to look at currently available and relevant policies, services and programs to make sure that my clients can access the services they need”, and “In the assessment session with the client, I empower the older person so that he/she can decide what to do, how to do it, and where to go for services”. Example items measuring the affective-based domain are, “Individuals who are different from me need to be respected”, “In my 8
daily work, I should treat all clients the same”, and “Diversity characteristics do not influence the care I give to my clients”. The full survey developed to evaluate learning outcomes for DT in community health and aged care can be found in supplementary file C. Administering the survey both pre and post the training intervention will provide a baseline to assess changes in perceptions and learning post the training (Kulik & Roberson, 2008). Behaviour Criteria Behaviour criteria is the degree to which trainees apply what they have learnt during the training to their jobs (D. L. Kirkpatrick, 1987). Assessing behavioural change has not been examined to the extent of other learning outcomes, such as attitudes or knowledge. Assessing this domain is more complex given that the measurement of this domain can rely on both subjective and more objective measures to accurately evaluate changes in behaviour following the training (Kalinoski et al., 2013). A review by Bezrukova et al. (2012) which critically assessed 178 articles on DT, identified that most measures of behavioural learning within the literature were short-term, using explicit measures (e.g. self-report surveys capturing conscious behavioural changes). Using more long term measures can assist in providing a better representation of actual changes in skills, compared to short term measures (Kulik & Roberson, 2008). Interviews can provide a useful addition to the evaluation by providing a nuanced understanding of participants’ views and outcomes with regard to the training. There has been widespread agreement that mixing qualitative and quantitative data collection methods is a useful way to collect a variety of data that can be used for the evaluation of training programs (Creswell & Clark, 2007; Greene, Benjamin, & Goodyear, 2001). The mixed methods approach allows researchers to capture additional information that has not been captured through other methods of data collection (Mertens, 2014) and is an effective way to assess various individual and organisational outcomes of the training (Bezrukova et al., 2012). As part of the DT for community health and aged care workers, an action plan is used to encourage changes in behaviour following the training. In the action plan, participants set themselves goals or priorities in order to change the way they deliver services to their clients. Interviews with the participants, conducted some duration after the training can assist in providing vital information about long-term changes in the way the workers deliver services to their clients, as a result of the training. The interviews follow up on the success, barriers and facilitators of carrying out their action plans hence evaluating behaviour change. Additionally, probing for other examples of change in onthe-job performance, examples of addressing diversity characteristics of clients, and whether learnings from the training have been applied to their jobs provides an evaluation mechanism for the behaviour criteria. Interviews can also tap into how the training assisted in allowing organisational mechanisms for supporting diversity to be set up (e.g. embedding diversity within professional development programs in the organisation, links to formal professional accreditation points, and the use of mentoring and peer support processes) (Horvat et al., 2014). A number of authors advocate that behavioural change requires practical information from real world situations that allows individuals to measure changed behaviours (Arthur Jr et al., 2003; King, Gulick, & Avery, 2010). Similarly, Cocchiara et al. (2010) state in relation to DT in a business setting, that assessing customer feedback is a way of assessing actual rather than perceived behavioural change. As such, it is vital to obtain client feedback from the aged care workers that self-report measures may not attain. Interviews with older people receiving care can directly assess behavioural changes of aged care workers following the DT, through evaluating feedback directly from clients
9
about how their care may have changed after the intervention. By obtaining and analysing direct client feedback, a more objective method of evaluation is gained (Bezrukova et al., 2012). Conducting interviews with clients taps into more implicit aspects of learning outcomes, by exploring changes in behaviours and attitudes from the perspectives of those who are receiving care (Kalinoski et al., 2013). Kalinoski et al. (2013) iterate the importance of using both implicit and explicit measures for evaluation since both these kinds of processes can be used to change implicit versus explicit aspects of attitudes and therefore behaviours. Conducting interviews with older people receiving care can not only provide an assessment of actual on-the-job changes, but also provide a third-party view on behavioural changes that may not be top of mind to the care providers. Information from those receiving care can augment the self-report information collected in the evaluation, by providing another perspective of behavioural and skill-based change from the training, and is of benefit to the complete evaluation process. Results The results category is the extent to which targeted outcomes occur as a result of the training and ensuing reinforcement (Kirkpatrick, 1987). Kirkpatrick and Kirkpatrick (2009) mention Return on Expectations (ROE) as the ultimate indicator of the value of the program, as they reflect the expectations of stakeholders (such as reference group members) which are turned into measurable success indicators. These success indicators are usually formulated in the training needs analysis by asking stakeholders “what will success look like to you?” (J. Kirkpatrick & Kirkpatrick, 2009). The reporting on these success indicators can inform whether targeted outcomes have occurred from the training. The success indicators may include whether:
Training has been delivered to the required number of people or organisations. Change in learning outcomes has occurred (cognitive, attitudinal and skilled-base outcomes) as measured by the survey Participants have achieved changes in behaviour through the development and implementation of an action plan (to implement an action for change in the organisation or in the way they deliver services to their clients), as captured through interviews with the aged care workers Change in service delivery has occurred, as captured through interviews with the clients of those who attended the training Participants were satisfied with their experience of the training, as assessed through the reaction measure.
Responding to the success indicators can help determine the effectiveness of the DT for community aged care workers. As such an evaluation framework for DT should incorporate measures for success indicators. An Evaluation Framework for the Community Aged Care Industry The revised Kirkpatrick model (Kirkpatrick and Kirkpatrick, 2009) was used to guide the key evaluation steps of a DT program directed towards the aged care industry. These steps have been collated in a framework (Figure 2). This framework is intended to inform a rigorous and systematic approach to training evaluation. The framework:
10
1. Begins with a training needs analysis to inform the design of the training and identify success indicators, critical behaviours, competencies and learning objectives that will be evaluated 2. Shows the design of a survey using a Delphi process 3. Includes an evaluation survey to be distributed prior to, and post training intervention, capturing learning outcomes 4. Suggests the training intervention be monitored by an independent observer to evaluate the reactions, content, structure and delivery of the workshop 5. Nominates that interviews with both participants of the workshop and clients of the participants, are conducted to assess behavioural change and any improvements in service delivery 6. Suggests that the findings are continuously analysed, training content adjusted, and ongoing iterative feedback is occurring through delivery of the program. Insert Figure 2 Discussion Diversity training in community aged care has the potential to deliver more appropriate care for older people. It has been shown in the literature, that appropriate care delivery may increase clients’ following of medical advice and generally provide more effective services to older people accessing aged care, ultimately enhancing their participation in healthcare (ACSA, 2015; Horvat et al., 2014).Typically, DT in health and aged care has been on culture or LGBTI training, focusing on one characteristic at a time. A DT program has been developed that introduces the premise of ‘diversity within diversity’, acknowledging that a person has more than one diversity characteristic, that these characteristics can intersect, and may be prioritised by individuals in different ways (Bagilhole, 2010; Hankivsky et al., 2014), consequently assisting or limiting their access to appropriate healthcare (Meyer et al., 2016). Evaluation studies assessing the effectiveness of DT programs in health and aged care have mainly used surveys assessing explicit learning outcomes from a single source, no long-term measures, or measures that look at behavioural outcomes (e.g. Spina, 2015; Pleasant et al., 2016). This paper has used Kirkpatrick and Kirkpatrick’s (2009) revised model of training and evaluation to guide the synthesis of the literature review findings according to the suggested five levels of evaluation: a training needs analysis, reaction criteria, learning criteria, behaviour criteria and results. The evaluation framework constructed in this paper is novel in the DT literature as it: 1. presents a framework specific to DT in community health and aged care that considers ‘diversity’ through a holistic rather than siloed lens; 2. develops a survey measuring cognitive, skill-based, and affective learning outcomes for DT in community health and aged care; 3. uses a mixed methods approach for evaluation; 4. evaluates behaviour change through using interviews with both health and aged care workers who participate in the training as well their clients; and 5. measures long term outcomes through the interviews conducted three months’ post workshop. The inclusion of the three learning outcomes proposed by Kraiger et al (1993) (cognitive, affective and skill-based learning outcomes) currently does not exist in diversity evaluation measures, with this paper taking the opportunity to develop a survey relevant to health and aged care workers, in order to measure these changes. Kalinoski et al. (2013) in their meta-analytic evaluation of DT outcomes, found that few studies focus on both cognitive and skill-based outcomes, with most research assessing affective outcomes through querying respondents’ attitudes and motivations following training. Within this paper, we uniquely propose the development of a survey that 11
measures all three learning outcomes, in order to provide more empirically sound and valid results on the effectiveness of DT workshops within the aged care sector. The use of a mixed methods approach in the framework provides additional strength for evaluation programs. Evaluation strength is limited by the use of a single source and self-report design, restricting the data to mainly subjective and explicit responses rather than a combination of subjective and objective and/or implicit and explicit findings (Holladay & Quiñones, 2008; Arthur, 2003). Mixed method evaluation adds rigour to the evaluation process and draw on the strengths of various data types (e.g. quantitative and qualitative data) ensuring a more comprehensive assessment of the training. Yet, surprisingly few evaluation programs within the diversity dominion use a mixed methods approach (Bezrukova et al., 2012), particularly the use of surveys combined with interviews and an independent observer as this framework proposes. In using a mixed methods approach, the inclusion of interviews in the evaluation process helps assess the behavioural changes following training and actual improvements in service delivery. Kulik and Roberson (2008) in their review of 74 studies on DT found a lack of attention devoted to assessing actual behavioural learning, with the few studies assessing this domain using mainly shortterm and self-report measures. Roberson, Kulik, and Pepper (2009) also state that changes in behaviour is the primary objective of DT programs, but is not reflected in most evaluation studies. Behaviour change following the training is particularly important to evaluate for DT in health and aged care, in order for workers to provide diversity appropriate care to their clients. In the proposed framework, an action plan is used in the DT to encourage behaviour change, and is evaluated using interviews with both health and aged care workers and their clients. The interviews also capture other changes in on-the-job performance and whether the training assisted in allowing organisational mechanisms for supporting diversity to be set up. Finally, this paper presents a formative framework that is relevant for the community health and aged care industry. With little documented training on diversity appropriate care existing in the literature, the effectiveness of this kind of training remains largely unknown. The framework constructed herein presents a unique model of evaluation specific to DT evaluation in the community aged care sector, and is relevant to the increasing need for training workers on diversityrelated issues. It is anticipated that the framework will allow for the development of effective DT programs by providing measurable indicators of their impact on learning, behavioural change and service delivery. This formative framework will be used to evaluate a DT workshop currently being delivered for community health and aged care workers; to measure whether the training assisted workers in providing more appropriate care to their clients. The survey tool and semi-structured interview guides for aged care workers and their clients will be validated in a subsequent study.
12
References ACSA. (2015). ACSA response to exposure draft legislation. Canberra: Aged and Community Services Australia. Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick's levels of training criteria: Thirty years later. Personnel psychology, 42(2), 331-342. Alvarez, K., Salas, E., & Garofano, C. M. (2004). An Integrated Model of Training Evaluation and Effectiveness. Human Resource Development Review, 3(4), 385-416. doi: 10.1177/1534484304270820 Arthur Jr, W., Bennett Jr, W., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in organizations: a meta-analysis of design and evaluation features. Journal of Applied Psychology, 88(2), 234. Australian Institute of Health and Welfare. (2015). Diversity in older Australians. Retrieved 18/09/2015, 2015, from http://www.aihw.gov.au/aged-care/diversity/ Bagilhole, B. (2010). Applying the lens of intersectionality to UK equal opportunities and diversity policies. Canadian Journal of Administrative Sciences / Revue Canadienne des Sciences de l'Administration, 27(3), 263-271. doi: 10.1002/cjas.167 Bezrukova, K., Jehn, K., & Spell, C. (2012). Reviewing diversity training: Where we have been and where we should go. Academy of Management Learning & Education, 11(2), 207-227. doi: 10.2307/23412323 Brinkerhoff, R. O., Brethower, D. M., Nowakowski, J., & Hluchyj, T. (2012). Program evaluation: A practitioner’s guide for trainers and educators (Vol. 2): Springer Science & Business Media. Brown, J. (2002). Training needs assessment: A must for developing an effective training program. Public Personnel Management, 31(4), 569-578. Celik, H., Abma, T., Klinge, I., & Widdershoven, G. (2012). Process evaluation of a diversity training program: The value of a mixed method strategy. Evaluation and Program Planning, 35(1), 54-65. doi: 10.1016/j.evalprogplan.2011.07.001 Chenowethm, L., Jeon, Y. H., Goff, M., & Burke, C. (2006). Cultural competency and nursing care: an Australian perspective. International nursing review, 53(1), 34-40. doi: 10.1111/j.14667657.2006.00441.x Cocchiara, F. K., Connerley, M. L., & Bell, M. P. (2010). “A GEM” for increasing the effectiveness of diversity training. Human Resource Management, 49(6), 1089-1106. doi: 10.1002/hrm.20396 Colton, D., & Covert, R. W. (2007). Designing and constructing instruments for social research and evaluation. San Francisco: John Wiley & Sons. Cook, C. T., Kosoko-Lasaki, O., & O'Brien, R. (2005). Satisfaction with and perceived cultural competency of healthcare providers: the minority experience. Journal of the National Medical Association, 97(8), 1078. Creswell, J. W., & Clark, V. L. P. (2007). Designing and conducting mixed methods research. Dalkey, N., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management science, 9(3), 458-467. Goldstein, I. L., & Ford, J. K. (2002). Training in Organizations (4th edition). Belmont, CA: Wadsworth. Gould, D., Kelly, D., & White, I. (2004). Training needs analysis: an evaluation framework. Nursing standard, 18(20), 33. Greene, J. C., Benjamin, L., & Goodyear, L. (2001). The merits of mixing methods in evaluation. Evaluation, 7(1), 25-44. Groot, W. (2000). Adaptation and scale of reference bias in self-assessments of quality of life. Journal of health economics, 19(3), 403-420. Hankivsky, O., Grace, D., Hunting, G., Giesbrecht, M., Fridkin, A., Rudrum, S., . . . Clark, N. (2014). An intersectionality-based policy analysis framework: Critical reflections on a methodology for advancing equity. International Journal for Equity in Health, 13(119).
13
Horvat, L., Horey, D., Romios, P., & Kis-Rigo, J. (2014). Cultural competence education for health professionals. Cochrane Database of Systematic Reviews(5). doi: 10.1002/14651858.CD009405.pub2. Hsu, C.-C., & Sandford, B. A. (2007). The Delphi technique: making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1-8. Johnson, D. W. (2008). Evaluating training. Thinking outside the borders, (17). http://www.library.illinois.edu/mortenson/book/08_wilcox.pdf Kalinoski, Z. T., Steele-Johnson, D., Peyton, E. J., Leas, K. A., Steinke, J., & Bowling, N. A. (2013). A meta-analytic evaluation of diversity training outcomes. Journal of Organizational Behavior, 34(8), 1076-1104. doi: 10.1002/job.1839 King, E. B., Gulick, L. M. V., & Avery, D. R. (2010). The Divide Between Diversity Training and Diversity Education: Integrating Best Practices. Journal of Management Education, 34(6), 891-906. doi: 10.1177/1052562909348767 Kirkpatrick, D. L. (1959). Techniques for evaluating training programs. Journal of ASTD, 11, 1-13. Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development: A guide to human resource development (2nd ed., pp. 18-27). New York: McGraw-Hill. Kirkpatrick, D. L. (1987). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: A guide to human resource development (3rd ed., pp. 301-319). New York: McGraw-Hill. Kirkpatrick, J., & Kirkpatrick, W. K. (2009). The Kirkpatrick four levels: A fresh look after 50 years 1959-2009 (pp. 1-13): The Kirkpatrick partners. Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78(2), 311-328. Kulik, C. T., & Roberson, L. (2008). Common goals and golden opportunities: Evaluations of diversity education in academic and organizational settings. Academy of Management Learning & Education, 7(3), 309-331. McGehee, W., & Thayer, P. W. (1961). Training in business and industry. New York: John Wiley and Sons. Mertens, D. M. (2014). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods. Los Angeles: Sage Publications. Meyer, C., Ogrin, R., Al-Zubaidi, H., Appannah, A., McMillan, S., Barrett, E., & Browning, C. (2017). Diversity training for community aged care workers: an interdisciplinary meta-narrative review. Journal of Educational Gerontology. In press. Pendry, L., Driscoll, D., & Field, S. (2007). Diversity training: Putting theory into practice. Journal of Occupational and Organizational Psychology, 80(1), 27-50. doi: 10.1348/096317906X118397 Pleasant, M. L., Molinari, V., Hobday, J. V., Fazio, S., Cullen, N., & Hyer, K. (2016). An evaluation of the CARES® Dementia Basics Program among caregivers. International Psychogeriatrics, 112. Roberson, L., Kulik, C. T., & Pepper, M. B. (2003). Using Needs Assessment to Resolve Controversies in Diversity Training Design. Group & Organization Management, 28(1), 148-174. doi: 10.1177/1059601102250028 Roberson, L., Kulik, C. T., & Pepper, M. B. (2009). Individual and environmental factors influencing the use of transfer strategies after diversity training. Group & Organization Management, 34(1), 67-89. Schneier, C. E. (1994). The training and development sourcebook: Human Resource Development. Spina, A. (2015). LGBTI Aged Care Training Evaluation Report: Aldo Spina Consultant. Figure Caption
14
Figures
Figure 1: The new Kirkpatrick model taken from Kirkpatrick and Kirkpatrick (2009)
15
Figure 2: An evaluation framework for diversity training in the aged care industry
16
S0149-7189(16)30192-6 http://dx.doi.org/doi:10.1016/j.evalprogplan.2017.03.007 EPP 1432
To appear in: Received date: Revised date: Accepted date:
5-9-2016 12-3-2017 21-3-2017
Please cite this article as: Appannah, Arti., Meyer, Claudia., Ogrin, Rajna., McMillan, Sally., Barrett, Elizabeth., & Browning, Colette., Diversity training for the community aged care workers: A conceptual framework for evaluation.Evaluation and Program Planning http://dx.doi.org/10.1016/j.evalprogplan.2017.03.007 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Title Diversity training for the community aged care workers: A conceptual framework for evaluation Funding This work was supported by the Department of Social Services, Australia. Authors Ms Arti APPANNAH (corresponding author)1 BachSocScience (Hons), MPsych (I/0), PhD Candidate 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
Email: [email protected] Dr Claudia Meyer 1 BachAppScience (Physio), MPubHealth, PhD 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
2
La Trobe University, Centre for Health Communication and Participation, Plenty Road & Kingsbury Drive, Bundoora, Victoria, 3086, Australia Email: [email protected] Phone: 61 3 8530 8105 Dr Rajna Ogrin1 BSc, BPod (Hons), PhD 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
2
University of Western Ontario, 151 Richmond St, Canada
3
University of Melbourne, Austin Health Department of Medicine, 145 Studley Rd, Heidelberg, Victoria, Australia Email: [email protected] Ms Sally McMillan1 GradCert (Anaphylaxis Management Training), Dip VocEducTrain, Registered Nurse 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
Email: [email protected] Ms Elizabeth Barrett1 GradCertCareerCoun, DipProjMan, DipPosPsych, GradDipHumanResources, MBusLeader 1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
Email: [email protected] Prof Colette Browning1 BSc (Hons), MSc, PhD 1
1
RDNS Institute, 31 Alma Rd, St Kilda, Victoria, 3182, Australia
2
Monash University, Wellington Rd & Blackburn Rd, Clayton, Victoria, Australia
3
International Institute for Primary Health Care Research, Shenzhen, China
Email: [email protected] Highlights
What this paper adds? As little is known about the effectiveness of diversity training in the community aged care industry, the construction of an evaluation framework provides an empirical and consistent method to assess the impact of this kind of training An existing training and evaluation framework is used to synthesise findings from the literature and construct a formative evaluation framework specific to diversity training programs in the community health and aged care industry The evaluation framework integrates three learning outcomes; cognitive, affective and skilledbased learning outcomes and presents a mixed methods approach for evaluating diversity training programs in the community aged care industry Abstract
Older Australians are an increasingly diverse population, with variable characteristics such as culture, sexual orientation, socioeconomic status, and physical capabilities potentially influencing their participation in healthcare. In response, community aged care workers may need to increase skills and uptake of knowledge into practice regarding diversity through appropriate training interventions. Diversity training (DT) programs have traditionally existed in the realm of business, with little research attention devoted to scientifically evaluating the outcomes of training directed at community aged care workers. A DT workshop has been developed for community aged care workers, and this paper focuses on the construction of a formative evaluative framework for the workshop. Key evaluation concepts and measures relating to DT have been identified in the literature and integrated into the framework, focusing on five categories: Training needs analysis; Reactions; Learning outcomes, Behavioural outcomes and Results. The use of a mixed methods approach in the framework provides an additional strength, by evaluating long-term behavioural change and improvements in service delivery. As little is known about the effectiveness of DT programs for community aged care workers, the proposed framework will provide an empirical and consistent method of evaluation, to assess their impact on enhancing older people’s experience of healthcare.
Keywords: Diversity training; Community care; Evaluation
2
Introduction By 2050, over 3.5 million older people in Australia will access community aged care services annually, with 80% of these services delivered in the community (ACSA, 2015). As Australians live longer, there will be increasing demands for aged care services to deliver more complex care in the community, and for an aged care workforce that can meet the needs of increasingly older populations from diverse backgrounds and with diverse needs. To ensure care provision is appropriate and leads to optimal health outcomes, the many characteristics that define an individual need to be considered in care assessment and planning. Further, there is growing acknowledgement that some populations are at risk of marginalisation and will require additional support within the aged care system to ensure equitable access and care; e.g. Aboriginal and Torres Strait Islanders; people from Culturally and Linguistically Diverse (CALD) backgrounds; people with dementia; people living in rural and remote areas; people experiencing financial disadvantage; people who are/or at risk of homelessness; and people who are lesbian, gay, bisexual, transgender and intersex (LGBTI) (Australian Institute of Health and Welfare, 2015). Therefore, training to raise awareness of diverse characteristics and their impact on access to healthcare becomes increasingly important for aged and healthcare workers. The need for delivering diversity training (DT) to age and healthcare workers has gained attention from policy makers, aged care services, professionals and educators as a strategy to improve outcomes in aged care; enhancing client participation in these services through diversity appropriate care (ACSA, 2015; Horvat, Horey, Romios, & Kis-Rigo, 2014). Diversity appropriate care is care in which professionals pay deliberate attention to the specific needs of an individual following from the interplay between characteristics that make up diversity in an individual, including but not limited to the broad groupings of gender, ethnic origin and socio-economic status (Celik, Abma, Klinge, & Widdershoven, 2012). Research on health disparities suggests that people who receive culturally sensitive care are likely to show an increased level of adherence to medical advice and to report greater satisfaction with their care providers (Chenowethm, Jeon, Goff, & Burke, 2006; Cook, Kosoko-Lasaki, & O'Brien, 2005). To extrapolate from this, it is possible that awareness of diversity appropriate care can yield similar outcomes to enhance an individuals’ experience of healthcare. An older person may have a number of diverse characteristics that may or may not impact upon their participation in healthcare. For example: an older person with low health literacy, who lives in a remote area, belongs to the LGBTI community and has multiple morbidities may have differing health needs from an older person with good health literacy, who lives in a metropolitan region and has similar multiple morbidities. Differing characteristics of a person can impact on appropriate access and delivery of health services. A term coined ‘intersectionality’ has been premised in the diversity literature; that humans are made up of more than a single characteristic, and that these characteristics are prioritised by individuals in different ways, and should not be viewed through siloed lenses, and, importantly, that their interplay requires special consideration (Bagilhole, 2010; Hankivsky et al., 2014). Healthcare professionals are required to recognise diverse characteristics in their clients, yet DT interventions to increase awareness of diversity appropriate care in the aged care industry are few. Awareness about diversity appropriate care in healthcare practices can be stimulated by providing health professionals with diversity awareness training (Celik et al., 2012), yet, to date this training has mainly been in the form of cultural competence training (Horvat et al., 2014). Other types of training have started to emerge in the literature such as Lesbian, Gay, Bisexual, Transgender and/or Intersex (LGBTI) training (Spina, 2015) or dementia care training for aged care workers (Pleasant et al., 2016). Yet these types of training again focus on one diversity characteristic through a siloed 3
lens, not discussing ‘diversity within diversity’ where people are made up of multiple diversity characteristics, some of which may have positive impacts on access to healthcare. Consequently, little research attention has been devoted to scientifically and systematically evaluating the effectiveness of DT programs (Pendry, Driscoll, & Field, 2007), particularly in the health and aged care industry. Indeed, Cocchiara, Connerley, and Bell (2010) acknowledge the inconsistent use (or non-use) of specific measurement criteria when evaluating DT, hence leading to difficulties in gauging the effectiveness of these types of interventions. Studies that have evaluated the outcomes of DT programs have predominantly used self-report measures collected from a single source, without assessing the long-term effects of learning outcomes (Bezrukova, Jehn, & Spell, 2012; Kulik & Roberson, 2008). Few DT programs for community health and aged care workers are documented in the literature, and thus little is known about the effectiveness of these programs. The evaluation of DT interventions is essential to measure their impact on the knowledge, skills and service delivery of health and aged care workers, as well as outcomes on clients (i.e. whether the training leads to healthcare that meets the needs of the client). A consistent and systematic approach to evaluating the outcomes of DT workshops can help inform their design and delivery, providing more effective services to older people accessing health and aged care and ultimately enhance their participation in health services. A DT program for community health and aged care workers has been developed and is underpinned by human rights, psychological and sociological theories (Meyer et al., 2016). The training assists participants to identify a broad range of diversity characteristics for older people in health and aged care, examines perspectives and assumptions related to diversity, and determines initiatives to improve service provision through the use of an action plan. The training is novel in community health and aged care since its focus is not solely on culture, and it moves away from a siloed view of diversity, addressing the ‘intersectionality’ of diversity characteristics in a person (Bagilhole, 2010; Hankivsky et al., 2014). The purpose of this paper is to draw from both the DT and evaluation literature to present an evaluation framework that is specific to DT for health and aged care workers and to comprehensively evaluate the developed training This paper reports on a narrative review of the literature, with findings synthesised according to revised categories in the Kirkpatrick model of training and evaluation (J. Kirkpatrick & Kirkpatrick, 2009). A framework specific to evaluating DT programs for community health and aged care workers is proposed, providing a consistent and empirically sound method of evaluation, to ultimately enhance training delivery and the provision of diversity appropriate care. Methodology for Literature Review and Synthesis A review and synthesis of the literature relevant to diversity programs and evaluation was conducted using business, sociology and health related electronic databases (SAGE Management and Organisational Studies; CINAHL; Informit Business; EMBASE; MEDLINE; Proquest Nursing and Allied Health; Proquest Psychology; ERIC; Proquest Health Management; Proquest Sociology; Business Source Complete; PsychINFO; and Proquest Business) and Google Scholar. The search of titles and abstracts was conducted from 1900s to July 2015, published in English, in order to identify the relevant training and evaluation literature, theories and models. The following search terms were used relevant to diversity: diversity; diversity theory; diversity model; diversity framework; and diversity training. The following search terms were used relevant to evaluation: evaluation; training evaluation; evaluation framework; evaluation methodologies and diversity training evaluation. All types of studies were included; randomised controlled trials, uncontrolled evaluations, qualitative research and descriptive pieces. 4
From 4599 articles screened by reading the title and abstracts of the papers, 28 papers relevant to DT and evaluation were found. The literature was reviewed and key concepts were identified relevant to the evaluation of DT. The concepts were then summarised into an all-inclusive, standardised framework for conducting an evaluation of a DT program within the aged care industry. A revised model of training and evaluation (Kirkpatrick and Kirkpatrick, 2009) depicted in Figure 1 was used to synthesise the findings of the review. This model of training and evaluation was based on an original model developed by Kirkpatrick (1959). The stages of the identified evaluation steps from the literature are discussed within the context of DT for the aged care industry, with relevant information included on instruments and methods of evaluation. Results Key steps for a formative evaluation framework for the community aged care sector has been identified through a review of the DT evaluation literature, using Kirkpatrick and Kirkpatrick’s (2009) model of training and evaluation. The findings have been categorised according to the following key components of the model: 1. training needs analysis; 2. reaction criteria; 3. learning criteria; 4. behaviour criteria; and 5. results. Each component will be discussed. Insert Figure 1 Needs analysis Kirkpatrick and Kirkpatrick (2009) identified training needs analysis as the first step to delivering and evaluating training, where critical behaviours, required competencies, learning objectives and necessary learning environments are established. According to Brown (2002), this training needs analysis should be an ongoing process where relevant data is gathered to help organisations determine whether their training objectives have been attained through the program. A needs analysis conducted prior to the training program can also provide a baseline assessment of the participants’ knowledge, assisting to direct the content of the workshop appropriately. Arthur Jr, Bennett Jr, Edens, and Bell (2003) purport that a thorough and methodical needs assessment is not only integral to the design and development of the training, but also to its evaluation by specifying a number of key features of the training for measuring the outcomes of the program (such as identifying what aspects of the training can be used to evaluate behavioural or learning outcomes). The training needs analysis is important for evaluation because unless informational needs are developed prior to the training, the measurement of the training effectiveness can be difficult (Brown, 2002). To assist in evaluation, Gould, Kelly, and White (2004) suggest that the outcomes of the training needs analysis should be clearly stated and matched to the original aims, organisational needs and requirements of the stakeholders. Roberson, Kulik, and Pepper (2003) also suggest that within a needs assessment, the strength and direction of trainees’ attitudes toward diversity, exposure to diversity-related issues (e.g. whether previous DT has been received), and potential inconsistencies between diversity-related attitudes and behaviours should be assessed. In undertaking a needs analysis, a three step process has been proposed by McGehee and Thayer (1961) which consists of an organisational analysis about where the training can be used in the organisation, an operations analysis to determine what should be included in the training, and a person analysis to find out about how employees are currently performing in the organisation. Similarly, Goldstein and Ford (2002) state that an organisational analysis is undertaken as a systemwide analysis beyond task and person, to determine the state of environmental conditions and 5
whether there are any barriers to the training. Resource requirements for the training and the availability of resources is needed as part of the organisational analysis (Goldstein & Ford, 2002) A needs analysis should consider the prior knowledge of the participants (Cocchiara et al., 2010), the organisational objectives and the characteristics of the task to be learned (Alvarez, Salas, & Garofano, 2004). Arthur Jr et al. (2003) adapts this process to suit a diversity context, proposing the three steps to be: 1. Organisational analysis: To determine support of diversity training in the organisation and organisational needs; 2. Task analysis: To understand the diversity training needs for employees to perform their job effectively; and 3. Person analysis: To decide on which individuals in the organisation require the training. To capture these tenets of information within a needs analysis, a questionnaire can be developed and sent to the trainees a few months prior to the workshop. According to the three-step process outlined above, the organisational analysis poses questions related to the organisation’s approach to diversity, current levels of organisational support and policy offered for the provision of inclusive services, and current processes in place in the organisation (e.g. diversity care plans and resources , to gauge the organisation’s investment in diversity related activities. For the task analysis, current staff knowledge on diversity, their understanding of diversity as a concept and how it can be applied in their day-to-day role as an aged care worker can be determined through the provision of a case study example of a client with diverse characteristics. The case-study allows employees to identify their perceptions of a client’s diverse characteristics, how these diverse characteristics may impact a client’s participation in healthcare, and what usual care entails for clients with diverse characteristics. Targeted behaviours to change for the training, learning objectives and success indicators (such as successfully developing and implementing an action plan relevant to diversity) can be identified in the task analysis. For the person analysis, the target audience for DT in the organisation is identified (e.g. both managers and frontline workers), including the optimal ratio of managers and frontline workers to include in the program so as to foster organisational support for a diversity approach in aged care. Reaction criteria Reaction criteria can be defined as the degree to which participants react favourably to the training (Kirkpatrick, 1987). Reaction criteria are usually operationalised through self-report, representing trainees’ affective and attitudinal (i.e. participants’ feelings towards the training) responses to the training program. Horvat et al. (2014) devised a four-dimensional conceptual framework for describing and assessing cultural competence in healthcare. While slightly different to DT, it expands on a pedagogical approach that includes the reactions or satisfaction with the teaching and learning methods used and the theoretical constructs that underpin it. Moreover, anecdotal evidence suggests that reaction measures are the most widely used evaluation criteria in applied settings (Arthur Jr et al., 2003). Drawing on the pedagogical approach proposed by Horvat et al. (2014), a survey consisting of items that assess satisfaction with the teaching and learning method, the practical applicability of the training content, whether the objectives of the training were attained, and their general satisfaction and enjoyment of the training delivery (Celik et al., 2012) can be used to evaluate reaction criteria. Example items that can be included in the survey are “I was satisfied with the way this program was delivered”, “I was able to meet the key learning objectives because of the way the training was presented”, and “I enjoyed participating in this workshop”. 6
Specific to DT in health and aged care, satisfaction items can also address whether the training aims and learning objectives were clear, whether the diversity training provided them with relevant and practical information for their jobs, and whether they felt the training on a whole helped them in their jobs. Example items are “the training key learning objectives were clearly defined”, “This diversity training did not provide me with information that was practical and that could be easily used in my daily work”, and “By participating in this training I improved my skills to more effectively meet the needs of our diverse community”. Satisfaction items developed to assess reactions to DT in community health and aged care are in supplementary file A. Independent observers can also assist in gauging reactions to the training and providing critical feedback for subsequent training workshops. For example, observers can make note of the variations in level and quality of discussions in the training, or a lack of engagement with a particular topic (Johnson, 2008). Using independent observers can enhance the validity of self-report measures, by rating the required knowledge and skill development (Kraiger, Ford, & Salas, 1993), the pedagogical methods used, the delivery and format of the workshop (Horvat et al., 2014), reactions to the training (D. L. Kirkpatrick, 1987) and participant engagement. Bezrukova et al. (2012) also state that the use of independent observers provide a more objective measure of the training, particularly in comparison to using solely self-report measures typically used for evaluating reactions. The use of forms and checklists can be used to capture information for the objectives of the evaluation program, assisting in providing formal criteria for the evaluation of training by the independent observer (Phillips, 1997). Types of factors to observe include the: extent to which training objectives appeared to be obtained (Colton & Covert, 2007); factors influencing training objectives attainment (including content, clarity of explanations, instructor’s knowledge of the subject, the use of training aids, appropriateness of the material to the group) (Horvat et al., 2014; Schneier, 1994); presence, absence, frequency or duration of participants’ behaviours as they occur (Brinkerhoff, Brethower, Nowakowski, & Hluchyj, 2012); and interpretive comments on the intentions or attitudes which might underlie the various observed behaviours (Celik et al., 2012). An independent observer rating sheet developed for DT training in community health and aged care can be found in supplementary file B. These guidelines can assist in capturing important information for the evaluation of a training program, while avoiding the biases that accompany more subjective evaluation measures (Groot, 2000). Learning criteria A conceptual basis for assessing learning outcomes from the training was first developed by Kirkpatrick (1976; 1987). He indicates that learning criteria evaluate the degree to which trainees acquire the knowledge, skills and attitudes from the training. There are some flaws with this model, most prominently the lack of clarity regarding what specific changes to learning may occur as a result of the training and what assessment techniques would be appropriate to measure these changes (Alliger & Janak, 1989; Kraiger et al., 1993). To address these concerns, Kraiger et al. (1993) advance Kirkpatrick’s (1987) model by defining the three categories of learning outcomes which help to explain the types of changes expected to occur following the training. These outcomes were specifically: 1. cognitive based (the knowledge gained from the training, including verbal knowledge and strategies); 2. skill-based (the behavioural changes resulting from the training); and 3. affective-based (attitudes, self-efficacy, and motivation in general). Kalinoski et al. (2013) contributed to the affective component of this model, proposing that
7
cognitive and skill-based outcomes may reflect relatively explicit outcomes (more conscious), but the affective-based outcomes better reflect both explicit and implicit (more unconscious) components. The content of the evaluation survey would need to include the following: 1. Cognitive-based domain: Questions that assess the types of knowledge acquired, how this knowledge can be applied and the skills developed through knowledge acquisition (Horvat et al., 2014). 2. The skills-based domain: Measures of behaviour and behavioural intention, together with questions on how healthcare workers adapt their service to suit the diverse needs of their clients, how they interact with their organisation to increase awareness about diversity, and how they share information with their client to increase their participation in diversity appropriate healthcare (Kalinoski et al., 2013). 3. The affective-based outcome domain: To contain items that measure the internal states that drive perception and behaviour. These include items that measure attitudes towards older people of diverse needs, perceptions of having diversity policies in an organisation, and feelings relating to dealing with diversity at work (e.g. enthusiastic, stressful, etc) (Kalinoski et al., 2013). For the evaluation of the three learning outcomes of a DT workshop for community aged care workers, to the best of the authors’ knowledge, there is currently no tool available. A DT evaluation survey has been developed to assess the learning criteria, using a Delphi technique. The Delphi technique, predominantly developed by Dalkey and Helmer in 1963, is a widely accepted method of eliciting and converging information from experts within a certain topic area. The method was devised in order to obtain the most reliable opinion agreement from a group of experts on a particular topic by subjecting them to a series of questionnaires, with controlled feedback on their opinions (Dalkey & Helmer, 1963). In round one, specific information about content areas of the topic is solicited from Delphi participants, which can include project and reference group members. In round two, the responses from round one are used to construct the survey instrument which is circulated to the Delphi panellists for further feedback. In the third round, the Delphi subjects receive the survey items and summarised feedback by the researcher, and are asked to revise their judgements in order to make further clarifications. In the final round, the list of remaining items, their ratings, opinions and items achieving consensus are distributed to the panellists, to provide a final opportunity to revise judgments (Hsu & Sandford, 2007). The developed survey measures the three learning outcomes of cognitive, skills and affective domains and will be validated in a subsequent study. Example items measuring the cognitive domain are, “I have the skills and knowledge to provide the right care to older people of diverse characteristics”, “The diverse characteristics of a client can be used positively to assist in their participation in healthcare”, and “In my work, I believe that I am capable of dealing appropriately with older people of diverse characteristics”. Examples of items measuring the skill-based domain are “When providing healthcare/community aged care services, I always appropriately adapt services in a way that includes the older person”, “I usually take the time to look at currently available and relevant policies, services and programs to make sure that my clients can access the services they need”, and “In the assessment session with the client, I empower the older person so that he/she can decide what to do, how to do it, and where to go for services”. Example items measuring the affective-based domain are, “Individuals who are different from me need to be respected”, “In my 8
daily work, I should treat all clients the same”, and “Diversity characteristics do not influence the care I give to my clients”. The full survey developed to evaluate learning outcomes for DT in community health and aged care can be found in supplementary file C. Administering the survey both pre and post the training intervention will provide a baseline to assess changes in perceptions and learning post the training (Kulik & Roberson, 2008). Behaviour Criteria Behaviour criteria is the degree to which trainees apply what they have learnt during the training to their jobs (D. L. Kirkpatrick, 1987). Assessing behavioural change has not been examined to the extent of other learning outcomes, such as attitudes or knowledge. Assessing this domain is more complex given that the measurement of this domain can rely on both subjective and more objective measures to accurately evaluate changes in behaviour following the training (Kalinoski et al., 2013). A review by Bezrukova et al. (2012) which critically assessed 178 articles on DT, identified that most measures of behavioural learning within the literature were short-term, using explicit measures (e.g. self-report surveys capturing conscious behavioural changes). Using more long term measures can assist in providing a better representation of actual changes in skills, compared to short term measures (Kulik & Roberson, 2008). Interviews can provide a useful addition to the evaluation by providing a nuanced understanding of participants’ views and outcomes with regard to the training. There has been widespread agreement that mixing qualitative and quantitative data collection methods is a useful way to collect a variety of data that can be used for the evaluation of training programs (Creswell & Clark, 2007; Greene, Benjamin, & Goodyear, 2001). The mixed methods approach allows researchers to capture additional information that has not been captured through other methods of data collection (Mertens, 2014) and is an effective way to assess various individual and organisational outcomes of the training (Bezrukova et al., 2012). As part of the DT for community health and aged care workers, an action plan is used to encourage changes in behaviour following the training. In the action plan, participants set themselves goals or priorities in order to change the way they deliver services to their clients. Interviews with the participants, conducted some duration after the training can assist in providing vital information about long-term changes in the way the workers deliver services to their clients, as a result of the training. The interviews follow up on the success, barriers and facilitators of carrying out their action plans hence evaluating behaviour change. Additionally, probing for other examples of change in onthe-job performance, examples of addressing diversity characteristics of clients, and whether learnings from the training have been applied to their jobs provides an evaluation mechanism for the behaviour criteria. Interviews can also tap into how the training assisted in allowing organisational mechanisms for supporting diversity to be set up (e.g. embedding diversity within professional development programs in the organisation, links to formal professional accreditation points, and the use of mentoring and peer support processes) (Horvat et al., 2014). A number of authors advocate that behavioural change requires practical information from real world situations that allows individuals to measure changed behaviours (Arthur Jr et al., 2003; King, Gulick, & Avery, 2010). Similarly, Cocchiara et al. (2010) state in relation to DT in a business setting, that assessing customer feedback is a way of assessing actual rather than perceived behavioural change. As such, it is vital to obtain client feedback from the aged care workers that self-report measures may not attain. Interviews with older people receiving care can directly assess behavioural changes of aged care workers following the DT, through evaluating feedback directly from clients
9
about how their care may have changed after the intervention. By obtaining and analysing direct client feedback, a more objective method of evaluation is gained (Bezrukova et al., 2012). Conducting interviews with clients taps into more implicit aspects of learning outcomes, by exploring changes in behaviours and attitudes from the perspectives of those who are receiving care (Kalinoski et al., 2013). Kalinoski et al. (2013) iterate the importance of using both implicit and explicit measures for evaluation since both these kinds of processes can be used to change implicit versus explicit aspects of attitudes and therefore behaviours. Conducting interviews with older people receiving care can not only provide an assessment of actual on-the-job changes, but also provide a third-party view on behavioural changes that may not be top of mind to the care providers. Information from those receiving care can augment the self-report information collected in the evaluation, by providing another perspective of behavioural and skill-based change from the training, and is of benefit to the complete evaluation process. Results The results category is the extent to which targeted outcomes occur as a result of the training and ensuing reinforcement (Kirkpatrick, 1987). Kirkpatrick and Kirkpatrick (2009) mention Return on Expectations (ROE) as the ultimate indicator of the value of the program, as they reflect the expectations of stakeholders (such as reference group members) which are turned into measurable success indicators. These success indicators are usually formulated in the training needs analysis by asking stakeholders “what will success look like to you?” (J. Kirkpatrick & Kirkpatrick, 2009). The reporting on these success indicators can inform whether targeted outcomes have occurred from the training. The success indicators may include whether:
Training has been delivered to the required number of people or organisations. Change in learning outcomes has occurred (cognitive, attitudinal and skilled-base outcomes) as measured by the survey Participants have achieved changes in behaviour through the development and implementation of an action plan (to implement an action for change in the organisation or in the way they deliver services to their clients), as captured through interviews with the aged care workers Change in service delivery has occurred, as captured through interviews with the clients of those who attended the training Participants were satisfied with their experience of the training, as assessed through the reaction measure.
Responding to the success indicators can help determine the effectiveness of the DT for community aged care workers. As such an evaluation framework for DT should incorporate measures for success indicators. An Evaluation Framework for the Community Aged Care Industry The revised Kirkpatrick model (Kirkpatrick and Kirkpatrick, 2009) was used to guide the key evaluation steps of a DT program directed towards the aged care industry. These steps have been collated in a framework (Figure 2). This framework is intended to inform a rigorous and systematic approach to training evaluation. The framework:
10
1. Begins with a training needs analysis to inform the design of the training and identify success indicators, critical behaviours, competencies and learning objectives that will be evaluated 2. Shows the design of a survey using a Delphi process 3. Includes an evaluation survey to be distributed prior to, and post training intervention, capturing learning outcomes 4. Suggests the training intervention be monitored by an independent observer to evaluate the reactions, content, structure and delivery of the workshop 5. Nominates that interviews with both participants of the workshop and clients of the participants, are conducted to assess behavioural change and any improvements in service delivery 6. Suggests that the findings are continuously analysed, training content adjusted, and ongoing iterative feedback is occurring through delivery of the program. Insert Figure 2 Discussion Diversity training in community aged care has the potential to deliver more appropriate care for older people. It has been shown in the literature, that appropriate care delivery may increase clients’ following of medical advice and generally provide more effective services to older people accessing aged care, ultimately enhancing their participation in healthcare (ACSA, 2015; Horvat et al., 2014).Typically, DT in health and aged care has been on culture or LGBTI training, focusing on one characteristic at a time. A DT program has been developed that introduces the premise of ‘diversity within diversity’, acknowledging that a person has more than one diversity characteristic, that these characteristics can intersect, and may be prioritised by individuals in different ways (Bagilhole, 2010; Hankivsky et al., 2014), consequently assisting or limiting their access to appropriate healthcare (Meyer et al., 2016). Evaluation studies assessing the effectiveness of DT programs in health and aged care have mainly used surveys assessing explicit learning outcomes from a single source, no long-term measures, or measures that look at behavioural outcomes (e.g. Spina, 2015; Pleasant et al., 2016). This paper has used Kirkpatrick and Kirkpatrick’s (2009) revised model of training and evaluation to guide the synthesis of the literature review findings according to the suggested five levels of evaluation: a training needs analysis, reaction criteria, learning criteria, behaviour criteria and results. The evaluation framework constructed in this paper is novel in the DT literature as it: 1. presents a framework specific to DT in community health and aged care that considers ‘diversity’ through a holistic rather than siloed lens; 2. develops a survey measuring cognitive, skill-based, and affective learning outcomes for DT in community health and aged care; 3. uses a mixed methods approach for evaluation; 4. evaluates behaviour change through using interviews with both health and aged care workers who participate in the training as well their clients; and 5. measures long term outcomes through the interviews conducted three months’ post workshop. The inclusion of the three learning outcomes proposed by Kraiger et al (1993) (cognitive, affective and skill-based learning outcomes) currently does not exist in diversity evaluation measures, with this paper taking the opportunity to develop a survey relevant to health and aged care workers, in order to measure these changes. Kalinoski et al. (2013) in their meta-analytic evaluation of DT outcomes, found that few studies focus on both cognitive and skill-based outcomes, with most research assessing affective outcomes through querying respondents’ attitudes and motivations following training. Within this paper, we uniquely propose the development of a survey that 11
measures all three learning outcomes, in order to provide more empirically sound and valid results on the effectiveness of DT workshops within the aged care sector. The use of a mixed methods approach in the framework provides additional strength for evaluation programs. Evaluation strength is limited by the use of a single source and self-report design, restricting the data to mainly subjective and explicit responses rather than a combination of subjective and objective and/or implicit and explicit findings (Holladay & Quiñones, 2008; Arthur, 2003). Mixed method evaluation adds rigour to the evaluation process and draw on the strengths of various data types (e.g. quantitative and qualitative data) ensuring a more comprehensive assessment of the training. Yet, surprisingly few evaluation programs within the diversity dominion use a mixed methods approach (Bezrukova et al., 2012), particularly the use of surveys combined with interviews and an independent observer as this framework proposes. In using a mixed methods approach, the inclusion of interviews in the evaluation process helps assess the behavioural changes following training and actual improvements in service delivery. Kulik and Roberson (2008) in their review of 74 studies on DT found a lack of attention devoted to assessing actual behavioural learning, with the few studies assessing this domain using mainly shortterm and self-report measures. Roberson, Kulik, and Pepper (2009) also state that changes in behaviour is the primary objective of DT programs, but is not reflected in most evaluation studies. Behaviour change following the training is particularly important to evaluate for DT in health and aged care, in order for workers to provide diversity appropriate care to their clients. In the proposed framework, an action plan is used in the DT to encourage behaviour change, and is evaluated using interviews with both health and aged care workers and their clients. The interviews also capture other changes in on-the-job performance and whether the training assisted in allowing organisational mechanisms for supporting diversity to be set up. Finally, this paper presents a formative framework that is relevant for the community health and aged care industry. With little documented training on diversity appropriate care existing in the literature, the effectiveness of this kind of training remains largely unknown. The framework constructed herein presents a unique model of evaluation specific to DT evaluation in the community aged care sector, and is relevant to the increasing need for training workers on diversityrelated issues. It is anticipated that the framework will allow for the development of effective DT programs by providing measurable indicators of their impact on learning, behavioural change and service delivery. This formative framework will be used to evaluate a DT workshop currently being delivered for community health and aged care workers; to measure whether the training assisted workers in providing more appropriate care to their clients. The survey tool and semi-structured interview guides for aged care workers and their clients will be validated in a subsequent study.
12
References ACSA. (2015). ACSA response to exposure draft legislation. Canberra: Aged and Community Services Australia. Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick's levels of training criteria: Thirty years later. Personnel psychology, 42(2), 331-342. Alvarez, K., Salas, E., & Garofano, C. M. (2004). An Integrated Model of Training Evaluation and Effectiveness. Human Resource Development Review, 3(4), 385-416. doi: 10.1177/1534484304270820 Arthur Jr, W., Bennett Jr, W., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in organizations: a meta-analysis of design and evaluation features. Journal of Applied Psychology, 88(2), 234. Australian Institute of Health and Welfare. (2015). Diversity in older Australians. Retrieved 18/09/2015, 2015, from http://www.aihw.gov.au/aged-care/diversity/ Bagilhole, B. (2010). Applying the lens of intersectionality to UK equal opportunities and diversity policies. Canadian Journal of Administrative Sciences / Revue Canadienne des Sciences de l'Administration, 27(3), 263-271. doi: 10.1002/cjas.167 Bezrukova, K., Jehn, K., & Spell, C. (2012). Reviewing diversity training: Where we have been and where we should go. Academy of Management Learning & Education, 11(2), 207-227. doi: 10.2307/23412323 Brinkerhoff, R. O., Brethower, D. M., Nowakowski, J., & Hluchyj, T. (2012). Program evaluation: A practitioner’s guide for trainers and educators (Vol. 2): Springer Science & Business Media. Brown, J. (2002). Training needs assessment: A must for developing an effective training program. Public Personnel Management, 31(4), 569-578. Celik, H., Abma, T., Klinge, I., & Widdershoven, G. (2012). Process evaluation of a diversity training program: The value of a mixed method strategy. Evaluation and Program Planning, 35(1), 54-65. doi: 10.1016/j.evalprogplan.2011.07.001 Chenowethm, L., Jeon, Y. H., Goff, M., & Burke, C. (2006). Cultural competency and nursing care: an Australian perspective. International nursing review, 53(1), 34-40. doi: 10.1111/j.14667657.2006.00441.x Cocchiara, F. K., Connerley, M. L., & Bell, M. P. (2010). “A GEM” for increasing the effectiveness of diversity training. Human Resource Management, 49(6), 1089-1106. doi: 10.1002/hrm.20396 Colton, D., & Covert, R. W. (2007). Designing and constructing instruments for social research and evaluation. San Francisco: John Wiley & Sons. Cook, C. T., Kosoko-Lasaki, O., & O'Brien, R. (2005). Satisfaction with and perceived cultural competency of healthcare providers: the minority experience. Journal of the National Medical Association, 97(8), 1078. Creswell, J. W., & Clark, V. L. P. (2007). Designing and conducting mixed methods research. Dalkey, N., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management science, 9(3), 458-467. Goldstein, I. L., & Ford, J. K. (2002). Training in Organizations (4th edition). Belmont, CA: Wadsworth. Gould, D., Kelly, D., & White, I. (2004). Training needs analysis: an evaluation framework. Nursing standard, 18(20), 33. Greene, J. C., Benjamin, L., & Goodyear, L. (2001). The merits of mixing methods in evaluation. Evaluation, 7(1), 25-44. Groot, W. (2000). Adaptation and scale of reference bias in self-assessments of quality of life. Journal of health economics, 19(3), 403-420. Hankivsky, O., Grace, D., Hunting, G., Giesbrecht, M., Fridkin, A., Rudrum, S., . . . Clark, N. (2014). An intersectionality-based policy analysis framework: Critical reflections on a methodology for advancing equity. International Journal for Equity in Health, 13(119).
13
Horvat, L., Horey, D., Romios, P., & Kis-Rigo, J. (2014). Cultural competence education for health professionals. Cochrane Database of Systematic Reviews(5). doi: 10.1002/14651858.CD009405.pub2. Hsu, C.-C., & Sandford, B. A. (2007). The Delphi technique: making sense of consensus. Practical Assessment, Research & Evaluation, 12(10), 1-8. Johnson, D. W. (2008). Evaluating training. Thinking outside the borders, (17). http://www.library.illinois.edu/mortenson/book/08_wilcox.pdf Kalinoski, Z. T., Steele-Johnson, D., Peyton, E. J., Leas, K. A., Steinke, J., & Bowling, N. A. (2013). A meta-analytic evaluation of diversity training outcomes. Journal of Organizational Behavior, 34(8), 1076-1104. doi: 10.1002/job.1839 King, E. B., Gulick, L. M. V., & Avery, D. R. (2010). The Divide Between Diversity Training and Diversity Education: Integrating Best Practices. Journal of Management Education, 34(6), 891-906. doi: 10.1177/1052562909348767 Kirkpatrick, D. L. (1959). Techniques for evaluating training programs. Journal of ASTD, 11, 1-13. Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development: A guide to human resource development (2nd ed., pp. 18-27). New York: McGraw-Hill. Kirkpatrick, D. L. (1987). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: A guide to human resource development (3rd ed., pp. 301-319). New York: McGraw-Hill. Kirkpatrick, J., & Kirkpatrick, W. K. (2009). The Kirkpatrick four levels: A fresh look after 50 years 1959-2009 (pp. 1-13): The Kirkpatrick partners. Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78(2), 311-328. Kulik, C. T., & Roberson, L. (2008). Common goals and golden opportunities: Evaluations of diversity education in academic and organizational settings. Academy of Management Learning & Education, 7(3), 309-331. McGehee, W., & Thayer, P. W. (1961). Training in business and industry. New York: John Wiley and Sons. Mertens, D. M. (2014). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods. Los Angeles: Sage Publications. Meyer, C., Ogrin, R., Al-Zubaidi, H., Appannah, A., McMillan, S., Barrett, E., & Browning, C. (2017). Diversity training for community aged care workers: an interdisciplinary meta-narrative review. Journal of Educational Gerontology. In press. Pendry, L., Driscoll, D., & Field, S. (2007). Diversity training: Putting theory into practice. Journal of Occupational and Organizational Psychology, 80(1), 27-50. doi: 10.1348/096317906X118397 Pleasant, M. L., Molinari, V., Hobday, J. V., Fazio, S., Cullen, N., & Hyer, K. (2016). An evaluation of the CARES® Dementia Basics Program among caregivers. International Psychogeriatrics, 112. Roberson, L., Kulik, C. T., & Pepper, M. B. (2003). Using Needs Assessment to Resolve Controversies in Diversity Training Design. Group & Organization Management, 28(1), 148-174. doi: 10.1177/1059601102250028 Roberson, L., Kulik, C. T., & Pepper, M. B. (2009). Individual and environmental factors influencing the use of transfer strategies after diversity training. Group & Organization Management, 34(1), 67-89. Schneier, C. E. (1994). The training and development sourcebook: Human Resource Development. Spina, A. (2015). LGBTI Aged Care Training Evaluation Report: Aldo Spina Consultant. Figure Caption
14
Figures
Figure 1: The new Kirkpatrick model taken from Kirkpatrick and Kirkpatrick (2009)
15
Figure 2: An evaluation framework for diversity training in the aged care industry
16