Approaching evaluation from a multilevel perspective: A comprehensive analysis of the indicators of training effectiveness

Human Resource Management Review xxx (xxxx) xxx–xxx

Contents lists available at ScienceDirect

Human Resource Management Review journal homepage: www.elsevier.com/locate/hrmr

Approaching evaluation from a multilevel perspective: A comprehensive analysis of the indicators of training effectiveness Traci Sitzmanna,⁎, Justin M. Weinhardtb a b

University of Colorado Denver, United States University of Calgary, Canada

AR TI CLE I NF O

AB S T R A CT

Keywords: Training evaluation Training effectiveness Multilevel framework Nomological network

We propose a multilevel framework that addresses the criteria that can be used to assess training effectiveness at the within-person, between-person, and macro levels of analysis. Specifically, we propose four evaluation taxa—training utilization, affect, performance, and financial impact—as well as the specific evaluation metrics that can be captured to examine the facets of each taxon. Our multilevel framework also clarifies the appropriate level of analysis for assessing each criterion variable and articulates when it appropriate to aggregate responses from a lower level of analysis to assess training effectiveness at a higher level of analysis. Finally, we illustrate how training evaluation criteria are interrelated because understanding constructs' nomological network is essential for gauging the depth of knowledge that can be inferred by any evaluation effort.

Donald Kirkpatrick delivered an address to the American Society of Training and Development in 1959 where he proposed a four levels approach to training evaluation—reaction, learning, behavior, and results (Kraiger & Ford, 2007). The levels were offered as a practical solution regarding how to evaluate training effectiveness and provided the valuable contribution of focusing evaluation efforts on measuring multiple outcomes as well as ensuring that contributions to business are considered as part of evaluation efforts (Bates, 2004). However, Kirkpatrick's framework is not grounded in theory and the assumptions of the model have been repeatedly disproven over the past 25 years (Alliger & Janak, 1989; Alliger, Tannenbaum, Bennett, Traver, & Shotland, 1997; Holton, 1996; Sitzmann, Brown, Casper, Ely, & Zimmerman, 2008; see Kraiger, 2002, pp. 333–335 for a critical review of Kirkpatrick's theory). Despite this fact, it is still the most frequently cited approach to training evaluation and continues to have a tremendous impact on research and practice. To propel the field beyond Kirkpatrick's levels of evaluation, we propose a multilevel training evaluation framework that examines the criteria that can be used to assess training effectiveness at the within-person, between-person, and macro levels of analysis. Specifically, we answer four questions that are imperative for promoting the science underlying evaluations of this crucial human resource function. Foremost, what outcomes should be assessed to appraise the effectiveness of an organization's overarching training initiative as well as individual training programs? To answer this question, we propose four evaluation taxa—training utilization, affect, performance, and financial impact—as well as the specific constructs that can be captured for each taxon. Although previous frameworks (e.g., Brown, 2005; Kirkpatrick, 1996; Kraiger, 2002; Kraiger, Ford, & Salas, 1993) have discussed one or more of the constructs in our taxonomy, we are the first to provide a comprehensive overview of the plethora of criteria that can be assessed to gauge training effectiveness. Second, what is the appropriate level of analysis for assessing training evaluation criteria? The past 50 years of training evaluation

⁎

Corresponding author. E-mail addresses: [email protected] (T. Sitzmann), [email protected] (J.M. Weinhardt).

http://dx.doi.org/10.1016/j.hrmr.2017.04.001 Received 25 July 2016; Received in revised form 30 March 2017; Accepted 2 April 2017 1053-4822/ © 2017 Elsevier Inc. All rights reserved.

Please cite this article as: Sitzmann, T., Human Resource Management Review (2017), http://dx.doi.org/10.1016/j.hrmr.2017.04.001

Decision to Enroll

WithinPerson

Complete vs. Drop Out of Training Lessons

Complete vs. Drop Out of Training

Attrition Rate

Episodic Satisfaction

Satisfaction

Episodic SelfEfficacy

Self-Efficacy

Training Reputation

Affective Indicators

Episodic Motivation

Motivation

Episodic Learning

Learning

Human Capital

Episodic Training Transfer

Training Transfer

Organizational and Team Performance

Performance Indicators

Personal Return on Investment

Return on Investment

Finanical Impact of Training

Fig. 1. Multilevel training evaluation taxonomy. Notes: Black lines indicate a compositional relationship among effectiveness indicators across levels of analysis; higher level constructs represent aggregates of lower level constructs and each lower level entity contributes equally to the higher level entity. Grey lines indicate a compilation relationship among effectiveness indicators across levels of analysis; lower level constructs combine in a nonlinear, complex manner to generate higher-level constructs. A dotted lines indicates that there is no direct compositional or compilational relationship for constructs across levels of analysis.

Which Employees Enroll

Enrollment Rate

Training Utilization

BetweenPerson

Macro

Levels of Analysis

Training Effectiveness

T. Sitzmann, J.M. Weinhardt

Human Resource Management Review xxx (xxxx) xxx–xxx

2

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

practice and research has fallen prey to a nearly uniform focus on the between-person level of analysis, ignoring the fact that examining within-person changes in learners is crucial for diagnosing training deficiencies and investigating macro-level criteria is essential for determining the overarching impact of training interventions. This practice is no longer acceptable; the training domain must expand beyond the between-person level to provide a comprehensive account of the range of factors that are affected by training initiatives as well as the dynamic nature of effectiveness criteria (Ployhart & Hale, 2014; Sitzmann & Weinhardt, in press). As part of this discussion, we will reveal several crucial discrepancies between theoretical conceptualizations and empirical operationalizations of training criteria that have hindered advancement in training evaluation. Third, when is it appropriate to aggregate responses from a lower level of analysis to assess training effectiveness at a higher level of analysis? Prolific strides have been made in clarifying that organizations are multilevel systems (e.g., Kozlowski & Klein, 2000; Ployhart & Moliterno, 2011). We rely on these conceptual and methodological paradigms to bridge the divide between micro and macro training evaluation criteria to ensure that research in this domain is no longer stunted by the cross-level fallacy (Ployhart & Moliterno, 2011). Finally, how are training evaluation criteria interrelated? Cronbach and Meehl (1955) articulated the necessity of understanding the nomological network of constructs that are subject to scientific inquiry. Capturing a breadth of effectiveness indicators is imperative for diagnosing why training may be ineffective and for maximizing its impact, but it is only by understanding the interrelatedness of evaluation criteria that we can gauge the depth of knowledge that can be inferred by a particular evaluation effort. Thus, we build on existing research (e.g., Alliger et al., 1997; Blume, Ford, Baldwin, & Huang, 2010; Colquitt, LePine, & Noe, 2000; Crook, Todd, Combs, Woehr, & Ketchen, 2011; Sitzmann et al., 2008) to provide a comprehensive account of the interrelations among evaluation criteria within and across levels of analysis. To answer these questions, we begin by presenting a multilevel training evaluation framework. We then elaborate on four evaluation taxa, the facets that comprise each taxon, and how the facets are related both across levels of analysis and to other evaluation criteria. 1. Overview of a multilevel training evaluation framework Our framework proposes that training evaluation and its predictors occur at multiple levels of analysis, including the withinperson, between-person, and macro levels (see Fig. 1). By levels of analysis, we are referring both to the hierarchical nature of constructs—such that employees are embedded in organizations, departments, and teams—and the temporal nature of processes—emphasizing that macro and within-person processes are not static phenomena. At the within-person level of analysis, training effectiveness evolves rapidly over time. For example, employees' understanding of the content and satisfaction with the material can vary substantially across training concepts. We use the term episodic to capture the dynamic nature of these processes. The between-person level of analysis represents stable comparisons across employees and is useful for determining which employees benefit from training. The macro level of analysis represents the overarching degree of training effectiveness across all employees and the benefits reaped for the organization, departments, teams, and other units within the organization. Macro processes also evolve over time, although these changes unfold slower than within-person processes (Kozlowski & Klein, 2000; Sitzmann & Weinhardt, in press). Fig. 1 summarizes four taxa that may be addressed by any evaluation effort and the specific criteria that can be assessed at the within-person, between-person, and macro levels of analysis. The first taxon represents training utilization. Training is a drain of organizational resources if few employees enroll and/or the majority of employees drop out before completing all program requirements. Furthermore, both enrollment and attrition can be assessed at all three levels of analysis, necessitating an in-depth understanding of how evaluation metrics will be utilized in order to target the appropriate level during data collection. The second taxon represents affective effectiveness indicators. Consistent with Brown (2005) and Kraiger et al. (1993), we classify satisfaction, self-efficacy, and motivation as affective outcomes, but we also extend beyond these theories by proposing that affective constructs should be monitored at both the within- and between-person levels of analysis. In addition, the reputation of training courses, programs, and the overarching training initiative should be examined at the macro level of analysis to gauge how training's notoriety may predispose a curriculum to fail or flourish. Third, training evaluation should target performance effectiveness indicators. Learning and training transfer have been extensively evaluated at the between-person level of analysis (Blume et al., 2010; Colquitt et al., 2000; Kraiger et al., 1993), but we extend beyond existing theories to note that these constructs also vary tremendously within-person over time. In addition, we discuss human capital and both organizational and team performance, which are macro performance effectiveness indicators. The final taxon addresses the financial impact of training. Financial evaluation efforts should spotlight the return on investment (ROI), which is a macro construct when focusing on the overarching organization and a between-person construct when focusing on employees. We rely on strides in multilevel theory to describe when it is appropriate to aggregate data collected at the within- or betweenperson levels of analysis to understand phenomena at higher levels of analysis. Specifically, we use the terms composition and compilation to describe the aggregation of constructs across levels of analysis (Kozlowski & Klein, 2000). Composition suggests that descriptive statistics at higher levels of analysis are aggregates of lower level effects, such that each lower level entity contributes equally to the higher-level entity. With composition, constructs are isomorphic, meaning that a construct is essentially the same thing at each level of analysis. For example, Simons and Roberson (2003) used aggregate justice perceptions to represent justice at the organizational level of analysis. Compilation suggests that lower level constructs combine in a non-linear, complex manner to generate higher-level constructs. 3

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

Thus, there is discontinuity in constructs across levels of analysis. For example, employees' learning cannot be aggregated to represent the organization's human capital (Ployhart, Call, & McFarland, in press). Rather, human capital represents a combination of employees' knowledge, skills, abilities, and other competencies (KSAOs), including KSAOs that employees' possessed before being hired as well as those acquired from training and other means. Therefore, it is inappropriate to aggregate individual-level learning to represent the overarching human capital of the firm (Ployhart et al., in press). Moreover, compilational effects require an extended timeframe to emerge, necessitating the use of a longitudinal design to detect these effects (Kozlowski, Brown, Weissbein, CannonBowers, & Salas, 2000). In Fig. 1, black lines connecting the evaluation constructs across levels of analysis illustrate compositional relationships whereas grey lines illustrate compilational relationships. A dotted lines indicates that there is no direct compositional or compilational relationship for constructs across levels of analysis. For example, there are moderators of the relationship between organizations' ROI and employees' ROI from training, such that we propose a null effect on average (for details see the Financial Impact section). Finally, the learning literature proposes that training teaches knowledge and skills that are relevant to employees' current jobs, whereas development assists employees in preparing for future jobs (Noe, 2008). However, the changing nature of work requires that employees constantly update their knowledge and skills if they want to remain in the workforce, blurring the boundary between training and development (Mathieu & Tesluk, 2010). Consistent with Mathieu and Tesluk, our multilevel framework focuses broadly on formal organizational learning initiatives. We use the term training throughout the manuscript and occasionally highlight how processes may differ across training and development programs. Next, we provide theoretical rationale regarding why each of the four taxa is an essential component of a comprehensive evaluation effort. Furthermore, we articulate how evaluation facets are related across levels of analysis. 2. Training utilization Although the number of training hours utilized affects the cost effectiveness of training (ASTD, 2013), training utilization has largely been neglected by evaluation theory. One of the primary objectives of this multilevel framework is to provide theoretical rationale for the processes that underlie the decision to utilize training resources, which employees utilize these resources, and the overall utilization rate. However, it is important to keep in mind that training utilization is only applicable in voluntary training; in mandatory courses, the training utilization rate should be close to 100%. 2.1. Training enrollment Training enrollment has a compositional relationship across the three levels of analysis. At the within-person level, enrollment represents the decision to enroll in training. At the between-person level, distinctions are made among employees to determine which employees enroll in training. At the macro level, the focus is on the overarching percentage of employees that enroll in training. The overarching enrollment rate affects the cost per learning hour used, such that it is only cost effective to create new or expensive training content if employees take advantage of learning opportunities (ASTD, 2013). The theory of planned behavior (Ajzen, 1985, 1991) elucidates whether employees take advantage of learning opportunities (Hurtz & Williams, 2009). Subjective norms and attitudes regarding training programs predict intentions to enroll, which is an immediate, proximal predictor of enrollment. These norms are a byproduct of organizations' training culture, suggesting that they originate at the macro level of analysis and have a top-down effect on employees (Sitzmann & Weinhardt, in press). The training culture affects whether training is prioritized and norms indicating that training is important increases the percentage of employees that utilize training resources. Self-efficacy is also positively related to goal establishment, such that employees who believe in their ability to succeed will be more likely to sign up for training and set higher learning goals (Bandura, 1989; Sun, Vancouver, & Weinhardt, 2014; Vancouver, More, & Yoder, 2008). The theory of planned behavior predicts training enrollment as long as employees have sufficient time and resources to engage in learning, there are few obstacles to training engagement, and there are not strong situational factors (e.g., sanctions) that compel the vast majority of employees to enroll (Hurtz & Williams, 2009; Sitzmann & Weinhardt, in press). Moreover, the overarching enrollment rate will be higher if the organization ensures that employees are aware of learning opportunities and provides employees with resources that promote learning (Hurtz & Williams, 2009). Yet, employees often have difficulty translating goals (i.e., enter training) into action due to competing work demands and personal constraints on their time (Schmidt & DeShon, 2007). Developing implementation intentions for enrolling in training and a plan for completing training are key interventions that can be implemented to enhance training utilization (Ajzen, 1991; Gollwitzer, 1999; Sitzmann & Johnson, 2012a). 2.2. Attrition from training Our multilevel framework conceptualizes attrition as electing not to complete training lessons (at the within-person level) or training programs (at the between-person level), and the ratio of employees who enroll but do not complete training (at the macro level). Thus, the within-person level provides the fine-grained detail regarding which lessons were completed by employees while the between-person level provides a summary of which employees completed training. Moreover, the within-person level can be aggregated to represent the percentage of training lessons completed. Attrition has a compositional relationship for the between-person and macro levels of analysis, such that the macro level attrition rate is an aggregate of the number of employees who complete training versus the number who drop out. However, the relationship is 4

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

compilational for the within- and between-person levels of analysis because employees may skip a lesson if they are already familiar with the material while still meeting all the program requirements. Thus, the partial attrition from skipping some of the lessons does not necessarily translate into dropping out of training. Attrition is a pervasive problem. Korn and Levitz (2013) examined the attrition rate from more than 9 million trainees enrolled in voluntary online courses and found that between 82 and 95% of trainees dropped out before completing training. Furthermore, less than half the registered apprentices in the United States construction industry complete the developmental program (Bilginsoy, 2003). Across seven offerings of voluntary online Microsoft Office training, between 70 and 91% of trainees dropped out before completing training (Sitzmann, 2012; Sitzmann & Ely, 2010; Sitzmann, Ely, Bell, & Bauer, 2010; Sitzmann & Johnson, 2012a, 2012b, 2014; Sitzmann & Wang, 2015). Despite its prevalence, attrition is a neglected topic in the training literature. Models of training effectiveness (e.g., Baldwin & Ford, 1988; Kirkpatrick, 1996; Kraiger et al., 1993) neglect to consider how attrition impacts the success of training. In empirical research, attrition is typically treated as a nuisance variable, such that employees who fail to complete a course are dropped from the statistical analyses (e.g., Sitzmann, Brown, Ely, Kraiger, & Wisher, 2009; Tannenbaum, Mathieu, Salas, & Cannon-Bowers, 1991). Moreover, the extant literature lacks an overarching theory to guide examinations of attrition, making it difficult to identify antecedents of this phenomenon. Dropping out may mean that employees are no longer interested in training and want to prioritize other goals or that they procrastinate in pursuing their learning goals and attrition occurs incidentally because training is never prioritized. Thus, attrition can be a functional or dysfunctional aspect of regulating multiple competing goals (Carver & Scheier, 1998), which is similar to how turnover can be functional or dysfunctional in organizations (Dalton, Krackhardt, & Porter, 1981). Dropping out is functional if employees are already knowledgeable about the topics covered or only need to learn a few of the skills taught in the program. Under these circumstances, employees can immediately learn the necessary information and then direct their time toward other activities. In contrast, attrition is dysfunctional if employees lack the knowledge and skills covered in training and failing to complete the program hinders their ability to effectively perform their job duties. Dysfunctional attrition may occur for a variety of reasons including lack of time to complete training, failing to incentivize employees who improve their skills, and deficiencies in self-regulated learning (Sitzmann & Weinhardt, in press). Failing to understand the causes and effects of training utilization has far reaching consequences for organizations, including the fact that resources are wasted if time and money are invested but employees do not complete training. Also, employees may lack the knowledge and skills necessary to effectively perform their job duties. Throughout our discussion, we will derive theoretical rationale regarding the role of enrollment and attrition in determining the overall effectiveness of organizational training programs. 3. Affective effectiveness indicators Extensive research has examined the role of affective effectiveness indicators—including satisfaction, self-efficacy, and motivation—in contributing to training success, such that multiple meta-analyses have examined the relationships among these variables and assessed their impact on performance effectiveness indicators (Alliger et al., 1997; Blume et al., 2010; Colquitt et al., 2000; Sitzmann & Ely, 2011; Sitzmann et al., 2008). We propose that research has overemphasized the between-person level of analysis, to the exclusion of examining how these constructs vary throughout training and assessing the role of the macro level of analysis—namely, the training reputation—in contributing to the success of training. 3.1. Satisfaction Satisfaction refers to subjective evaluations of the training experience (Brown, 2005). Course satisfaction research has uniformly been conducted at the between-person level of analysis, and we are only aware of three studies that have examined satisfaction at the within-person level of analysis (Sitzmann & Johnson, 2014; Sitzmann, Song, & Wang, 2017; Sitzmann et al., 2009). We advocate monitoring changes in satisfaction over time, rather than routinely assessing satisfaction at the end of training, and striving for a more limited role of satisfaction in evaluation endeavors. Foremost, within-person research reveals that course satisfaction does not have a significant effect on learning; rather, learning predicts satisfaction, which directly contradicts Kirkpatrick's (1996) framework (Sitzmann et al., 2017). In addition, performance feedback—which is inherently confounded with learning in the vast majority of research—is primarily responsible for the satisfaction/learning relationship (Sitzmann et al., 2017). Moreover, Uttl, White, and Gonzalez (2016) conducted a meta-analysis and found teaching evaluations only overlap with 1% of the variance in learning. Thus, designing courses to maximize satisfaction will not enhance learning. Kirkpatrick's levels of evaluation has resulted in an overreliance on satisfaction surveys and potentially designing training to maximize student enjoyment. It is a well-known principle that employees behave in a manner to receive favorable performance evaluations, often to the exclusion of behaviors that are not part of the evaluation (Kerr, 1975). Thus, if instructors are evaluated with satisfaction surveys, the program design may emphasize entertainment to the exclusion of both requiring the arduous effort that is essential for knowledge acquisition and providing the negative feedback that may be necessary for ensuring that employees progressively make strides toward content mastery (Sitzmann et al., 2017). The choice of evaluation metrics employed may explain in part the lament of many employers that employees frequently do not transfer knowledge to the work environment (Baldwin & Ford, 1988). The overreliance on satisfaction evaluation metrics is an excellent example of the folly of hoping for learner achievement while rewarding instructors who create satisfying instructional experiences. It is also important to acknowledge that there is a compilational relationship between satisfaction at the within- and between5

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

person levels of analysis. Employees do not weigh their satisfaction with each lesson equally when completing an overarching satisfaction measure; rather, extreme dissatisfaction at a single point in time may receive undue weight, lowering post-training assessments even if employees enjoyed the vast majority of the instructional experience. This is consistent with a snapshot view of affect, such that salient and extreme affective moments are weighted heavily and the vast majority of the affective episode receives zero weight when rating an overall affective experience (Fredrickson & Kahneman, 1993). Thus, the duration of positive and negative affect is neglected when individuals retrospectively evaluate the overarching episode so post-training assessments will not provide the level of detail necessary to enhance satisfaction for future trainees. Assessing satisfaction at the between-person level of analysis is also misaligned with affect theory, which suggests that emotional states vary tremendously over time and across performance episodes (Beal, Weiss, Barros, & MacDermid, 2005). Moreover, it relies on Kirkpatrick's disproven assumption that favorable satisfaction ratings are a precursor to learning (Alliger et al., 1997; Kirkpatrick, 1996; Sitzmann et al., 2008, 2017). Assessing satisfaction at the between-person level of analysis precludes evaluating the dynamic interplay among constructs and fails to consider that learning and performance feedback affect course satisfaction, rather than satisfaction affecting learning (Sitzmann et al., 2017). 3.2. Self-efficacy Self-efficacy is defined as belief in one's ability to perform a given task (Bandura, 1977). Kraiger et al.’s (1993) theory of learning outcomes, proposes that overreliance on cognitive and behavioral assessments while omitting affective learning provides an incomplete profile of the effectiveness of training. Thus, affective effectiveness indicators—including self-efficacy—should be conceptualized as indicators of learning, rather than as precursors to traditional learning assessments. They also advocate that improving affective outcomes should be the target of a breadth of training programs, including those designed to improve creativity, organizational commitment, safety, and tolerance for diversity. This has led to the recommendation that “Training should be designed to promote self-efficacy and then to reinforce it afterward” (Salas, Tannenbaum, Kraiger, & Smith-Jentsch, 2012, p. 84; emphasis added by Salas et al.). Yet, the self-regulation literature suggests that artificially inflating trainees' self-efficacy may impair performance effectiveness indicators. Salas et al.’s (2012) recommendation was based on research conducted at the between-person level of analysis, whereas within-person research cautions against artificially inflating employees' self-efficacy. Specifically, a plethora of research at the between-person level has established a positive relationship between self-efficacy and both learning and training transfer (for metaanalyses of the literature see Blume et al., 2010; Colquitt et al., 2000; Sitzmann & Ely, 2011). However, research conducted at the within-person level is essential for disentangling the dynamic interplay between self-efficacy and performance effectiveness indicators (Sitzmann & Yeo, 2013; Vancouver, 2005; Vancouver et al., 2013). Training represents a preparatory context—its explicit purpose is to increase employees' competencies so that they have the knowledge and skills necessary to perform work tasks—and both control and self-efficacy theories caution that high self-efficacy may impair performance in preparatory contexts (Bandura, 1977; Vancouver & Kendall, 2006). The issue at hand is that employees rely on their self-efficacy to determine the level of resources necessary for attaining adequate performance. When employees are confident in their ability to succeed, they reduce resource allocation (Vancouver & Kendall, 2006). If this confidence is unwarranted, it results in failing to allocate sufficient resources toward knowledge acquisition and self-efficacy having a negative within-person effect on performance. Thus, designing training to artificially enhance self-efficacy may set employees up for failure, such that they leave training with excessive confidence in their ability to succeed while lacking the skills necessary to competently perform work tasks. Evaluating between-person self-efficacy may lead to erroneous assumptions about employees' competencies, and self-efficacy should only be evaluated at the between-person level if it is assessed in conjunction with performance effectiveness indicators to detect if employees' overconfidence may be setting them up for failure. Moreover, a meta-analysis of within-person research revealed that self-efficacy is an indicator of past performance rather than the driving force affecting future performance (Sitzmann & Yeo, 2013). Thus, we recommend a limited role of self-efficacy in within-person evaluations (see the discussion of training enrollment for an example regarding when self-efficacy is an invaluable tool for enhancing training effectiveness). Further, researchers should avoid the cross-level fallacy—it is inappropriate to assume that self-efficacy has similar relationships at the within- and between-person levels of analysis because this effect is compilational. 3.3. Motivation Motivation refers to “the direction, intensity, and persistence of learning-directed behavior in training contexts” (Colquitt et al., 2000, p. 678). Similar to the aforementioned affective effectiveness indicators, we propose a compilational effect for motivation at the within- and between-person levels of analysis. Employees may not weigh their motivation across each temporal period equally when evaluating their post-training motivation, but both within- and between-person assessments should play a role in evaluation endeavors. Assessing motivation pre- and post-training at the between-person level of analysis is valuable for detecting which employees are ready for training and have the desire to transfer training to the job (Noe, 2008). Further assessments can then be used to detect why motivation is lacking and interventions can be implemented to enhance the desire to learn and transfer whenever necessary. At the within-person level of analysis, temporal patterns of motivation should be monitored because remaining motivated is essential for allocating resources toward training and, thus, learning a content domain (Beal et al., 2005; Kanfer & Ackerman, 1989). Individuals have a limited pool of attentional resources that can be allocated toward the task at hand, self-regulation of attention, and 6

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

off-task attentional demands. Maximizing on-task focus and self-regulated learning while simultaneously minimizing off-task focus is imperative for knowledge acquisition. Monitoring motivation at the within-person level of analysis may serve as an essential signal revealing when employees no longer have the attentional focus required for learning, necessitating the use of breaks or changes in the current activity to facilitate reengagement. 3.4. Training reputation The training reputation has received only brief mention from previous theories of training effectiveness (Brown, 2005), despite the fact that it is a vital barometer of macro-level affective evaluations. Similar to an organization's reputation (Lange, Lee, & Dai, 2011), the training reputation should be approached from a multidimensional perspective. Specifically, we propose that the training reputation should be evaluated from the perspective of both the organization's employees and external constituents and begin by discussing employees' perspective. Lange et al. (2011) proposed that organizations' reputation is comprised of three dimensions: being known, being known for something, and generalized favorability. Regarding training, employees' first perception of the reputation is knowing that training exists. They then adjust their perceptions as they acquire information about training (e.g., training targets a key organizational initiative). Finally, employees' perceptions change to reflect the generalized judgment of those who have participated in training. Employees form expectations about training before starting a course based on experiences with previous courses and comments from colleagues (Facteau, Dobbins, Russell, Ladd, & Kudisch, 1995). If employees perceive that training is a waste a time, they may lack the desire to enroll and exert effort in training, regardless of the actual course quality (Switzer, Nagy, & Mullins, 2005). Thus, maximizing the success of any training initiative should begin by monitoring the training reputation and making improvements whenever necessary to correct employees' perceptions of deficiencies. The training reputation must also be examined from the perspective of external constituents. The training reputation originates within the organization, but attaining training awards (e.g., ATD Best Award) as well as acknowledgements in publications and other public forums can derive an organizations' training reputation among external constituents. Organizations can also market their key training initiatives to enhance their training reputation (Facteau et al., 1995; Kraiger, 2002). One way that other organizations judge the reputation of training is through course satisfaction assessments, and human resource professionals believe these assessments provide valuable information about the quality of training (ASTD, 2013; Harman, Ellington, Surface, & Thompson, in press; Kraiger, 2002). From an institutional theory perceptive, the training reputation affects how other organizations adopt training (Strang & Macy, 2001; Strang & Meyer, 1993; Strang & Soule, 1998). Mimicking another organization's training happens for two reasons: economic (Katz & Shapiro, 1987) and legitimacy (DiMaggio & Powell, 1983), and these reasons can work in concert (Kennedy & Fiss, 2009). Thus, organizations look outward to other firms' training reputation when creating their own training initiatives, and the reputation that is sent out to other organizations emerges through the perceptions of an organization's current employees. In summary, we propose a compilational and reciprocal relationship for employees' satisfaction, self-efficacy, and motivation with their perceptions of the training reputation. The perceptions of various employees as well as the three between-person constructs are not weighted equally when establishing the training reputation, but employees who leave training feeling dissatisfied, demotivated, and doubtful in their ability to succeed will impair a program's reputation while employees with the opposite experience will enhance a program's prestige. Moreover, some employees' perceptions will be greatly swayed by the training reputation whereas others will be blissfully unaware of the training reputation. Employees should derive greater satisfaction with training, be more motivated, and have greater confidence in their ability to succeed when they are aware that training has a favorable reputation. 4. Performance effectiveness indicators Performance effectiveness indicators focus on changes in learning and transfer at the within-person level of analysis as well as between-person differences in these constructs. The success of a training initiative is enhanced when these between-person differences translate into increased human capital and improved organizational and team performance. 4.1. Learning Learning is defined as “a relatively permanent change in knowledge or skill produced by experience” (Weiss, 1990, p. 172). Kraiger et al. (1993) provide a comprehensive account of the learning construct domain and the types of assessments that can be used to assess improvements in cognitive and skill-based knowledge. We advance beyond their theoretical framework by distinguishing between learning and learning performance, articulating how learning is related at the within- and between-person levels of analysis, and by clarifying when evaluation efforts should concentrate on episodic or post-training assessments. Foremost, existing research has confounded learning—defined as content mastery—with learning performance, which reflects a combination of learning, prior knowledge, and the difficulty of the performance assessment (Sitzmann et al., 2017). Instructors can inflate performance feedback by administering easy performance assessments so research should partial out the variance in learning performance that is due to performance assessment difficulty and past performance to detect true relationships with learning. In addition, there is a compilational relationship between understanding training lessons (i.e., episodic learning) and overall content knowledge (i.e., between-person learning). Thus, it is inappropriate to average scores across learning assessments to compare employees' overarching understanding of the content domain. Rather, some knowledge and skills make a greater contribution to 7

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

acquiring expertise than others and scores on assessments of these skillsets should receive greater weight when comparing learning across employees. Learning assessments should be incorporated in training evaluation efforts if the purpose of the evaluation is to provide employees with feedback or to make decisions regarding program retention, program revisions, and instructor retention (Kraiger, 2002). Withinperson evaluation efforts require substantially more effort, but the extra resources are worth it if the assessments are used to provide employees with extensive feedback and to alter training on an impromptu basis to ensure mastery of key learning objectives. For example, adaptive guidance relies on performance assessments to council employees on where they should focus subsequent resources (Bell & Kozlowski, 2002). Altering the learning pathway based on episodic evaluation efforts substantially enhances knowledge and skill acquisition. Between-person evaluations assess learning at the end of training when it is too late to enhance the content mastery of the current group of employees. Yet, these assessments provide sufficient information to make decisions regarding instructor retention and to compare employees. Thus, the level of detail desired by the evaluation effort should drive decisions regarding whether learning should be assessed at the within- or between-person level of analysis. 4.2. Human capital Human capital is defined as the aggregate of employees' KSAOs and is an essential component of organizations' competitive advantage (Kim & Ployhart, 2014; Noe, Clarke, & Klein, 2014; Ployhart & Hale, 2014). Human capital provides organizations with a competitive advantage because it is expensive to acquire, difficult to imitate, and empowers the capacity for action (Ployhart & Moliterno, 2011; Ployhart et al., in press). Becker's (1964) human capital theory provides a theoretical foundation for justifying training expenses. Specifically, Becker distinguished between generic and firm-specific human capital. Generic human capital includes general skills, cognitive ability, motivation, and other individual differences that are not unique to an organization and are developed via prior experiences and education. In contrast, firm-specific human capital consists of knowledge and skills that are unique to a given job or organization and must be developed by the organization. Investing in firm-specific human capital represents an initial upfront cost, but these expenses are typically recovered when productivity increases because employees' skills are better than the organization's competitors (Kim & Ployhart, 2014). However, this is predicated on the assumption that employees remain with the firm. When training increases employees' skills, some of that value is retained by the employees themselves and some is captured by the organization (Ployhart et al., in press). When employees attain substantial value, the value may be lost to the organization if an employee becomes more marketable and leaves for another job. The majority of research on human capital has been conducted by economists examining the effect of training initiatives on national economic performance (Aguinis & Kraiger, 2009). This body of literature reveals that training improves the quality of the workforce, which is one of the strongest predictors of national economic growth (Aguinis & Kraiger, 2009; Becker, 1964). However, organizations also reap the benefits of firm-specific knowledge and skills since it enhances the rate with which employees learn their job duties, increases productivity, and is difficult for the organization's competitors to imitate (Aguinis & Kraiger, 2009; Barney, 1996; Hatch & Dyer, 2004; Kim & Ployhart, 2014). The relationship among between-person learning and human capital is compilational because human capital is not simply an aggregate of individual differences in knowledge. Rather, it is an emergent process influenced by interactions between individual differences and the work environment (including policies that enhance or inhibit increases in knowledge and skills), suggesting that human capital emergence must be examined from a cross-level perspective (Noe et al., 2014; Ployhart & Moliterno, 2011). Moreover, human capital is affected by employee selection, informal learning, knowledge sharing, and a host of factors beyond training. 4.3. Training transfer Training transfer is defined as the extent to which the knowledge and skills acquired in training are applied across settings and situations over time (Blume et al., 2010). At the between-person level of analysis, this distinction represents comparisons across employees in terms of who applies trained concepts and maintains the change over time. At the within-person level of analysis, episodic transfer represents successive attempts to utilize training content on the job, which is valuable for detecting when knowledge and skills are utilized post training. Similar to the learning relationship discussed earlier, some skills contribute to expertise more than others, representing a compilational effect for episodic and between-person transfer. It is often suggested that the rate of training transfer is abysmal, and the estimate that about 10% of training transfers has been touted as fact since the 1980s (Baldwin & Ford, 1988; Detterman, 1993; Ford, Yelon, & Billington, 2011; Georgensen, 1982; Holton & Baldwin, 2003). Yet, measuring transfer is challenging, precluding capturing precise indicators of the extent to which employees maintain trained skills and generalize them across applicable tasks (Ford et al., 2011). Overcoming this challenge is crucial because gauging the role of transfer in improving job performance is the “paramount concern of training efforts” (Baldwin, Ford, & Blume, 2009, p. 41). Existing theories and research uniformly focus on transfer at the between-person level of analysis, but adopting a multilevel approach may prove instrumental for understanding the factors that affect transfer. At the within-person level of analysis, it is important to gauge whether employees self-regulate training transfer. Self-regulation of transfer refers to striving to apply the knowledge and skills learned in training to other domains via control over affective, cognitive, and behavioral processes (Sitzmann & Ely, 2011). Training is rarely so comprehensive that it can advance employees' knowledge from novice to expert status 8

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

so employees must self-regulate their transfer and engage in informal learning if they are going to acquire expertise in a domain. At the macro level, the training culture has a top-down effect on training transfer (Sitzmann & Weinhardt, in press). Supervisor and peer support is imperative for ensuring that the workplace accommodates trained knowledge and skills and that employees have assistance when applying these insights on the job (Baldwin & Ford, 1988; Noe, 1986). 4.4. Organizational and team performance The primary goal of training initiatives is to increase performance at higher levels of analysis, including teams, departments, and the overarching organization (Kozlowski et al., 2000). Macro performance indicators include productivity, sales, revenue, market value, shareholder return, and overall profitability (Aguinis & Kraiger, 2009; Ployhart & Hale, 2014). Providing internal training to a greater percentage of employees positively affects firm-level financial performance (Kim & Ployhart, 2014), and team training is valuable for improving cognitive, affective, and performance outcomes as well as teamwork processes (Salas et al., 2008). The between-person variable that is aligned with macro level performance is employees' job performance, and the effect of training transfer on organizational performance is indirect via job performance (Ployhart et al., in press). However, the expectation that job performance can be aggregated to represent organizational performance is too simplistic because a host of factors—including an organizations' competitors, customer demand, human resource management functions (e.g., selection, compensation), and alignment between these functions—have a substantial impact on a firm's competitive advantage (Aguinis, 2009; Delaney & Huselid, 1996; Ployhart & Hale, 2014). Although training is a small piece of the broader human resource management function, small effects can have large financial consequences for organizations (Morrow, Jarrett, & Rupinski, 1997). Kozlowski et al. (2000; Kozlowski & Salas, 1997) provide a theoretical overview regarding how performance at lower levels of analysis combine to affect team and organizational performance. Foremost, employees must engage in horizontal transfer, meaning that they must apply training content across settings and contexts. In addition, vertical transfer must occur, meaning that individuallevel training outcomes must have an effect on outcomes at higher levels in the organization. The researchers rely on the example of total-quality management (TQM) to illustrate the processes that affect horizontal and vertical transfer (Kozlowski et al., 2000). The goal of TQM is to continually improve products, processes, and services. However, topdown contextual factors—including leadership, rewards, and the job design—inhibit horizontal transfer when the context is misaligned with training. For example, insufficient opportunities to practice newly acquired skills and a lack of leadership support reduce the extent to which training content is applied on the job. Moreover, TQM programs often deliver the same content to most members of the organization, which relies on the assumption that the contributions of employees sum to yield higher-level effects. However, this additive model often fails to represent reality, whereby quality reflects a complex combination of different contributions across employees. This renders TQM programs inefficient at best and more likely ineffective. Promoting vertical transfer requires determining the distinct knowledge and skills needed for employees in various positions and at different levels in the organization as well as how skills blend across employees. As this example illustrates, we propose a compilational relationship between training transfer and both team and organizational performance (see Fig. 1). 5. Financial impact We recommend gauging the financial impact of training by assessing ROI—which is an indicator of training costs relative to the benefits (Phillips & Phillips, 2007). We focus on ROI because it is directly affected by training and is less affected by factors outside of training (e.g., market conditions) than other financial indicators (e.g., organizational profit and return to shareholders). ROI can be assessed at both the macro and individual levels of analysis. Although some programs and lessons have a greater impact on employees' skillsets than others, the financial impact is typically only realized once an entire program of instruction has been completed. As such, personal ROI should be examined at the between-person rather than the within-person level of analysis. 5.1. Personal return on investment Personal ROI reflects the individuals' cost-benefit ratio for participating in training and development. Development that leads to a job promotion and a salary increase enhances the personal ROI. However, most training assists employees in performing their current job duties. High job performance as a result of skill proficiency attained via training may be rewarded with merit raises and bonuses, but these financial incentives tend to be less than those received for a promotion. Thus, employees may not always receive a personal financial benefit from training. Yet, on the cost side of the equation, completing training is invaluable for ensuring that employees do not lose their jobs. Employees also become more useful and adaptable by completing training, which may not be quantifiable. Thus, personal ROI can be assessed objectively or based on employees' perceptions. For training and development to produce a positive personal ROI, learning must be the most lucrative use of employees' time. If employees could contribute more to firm performance (as well as their own bottom line) by completing their job duties and could acquire essential skills on the job, devoting time to training could adversely affect personal and firm ROI. Thus, the financial cost of devoting time to training and development is crucial for accurately computing ROI. The most pervasive development programs in the United States are college degree programs. The ROI for attaining a college degree was over 350,000 dollars in 2012 (Lavelle, 2012). This statistic reflects the difference in median lifetime earnings of college and high school graduates while subtracting the cost of college tuition, room and board, and books. Moreover, the earning gap between those with high school and college degrees has more than doubled in the last 30 years and this trend can be attributed to a 9

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

Levels of Analysis Organizational & Team Performance

Attrition Rate +

-

Macro

Enrollment Rate

+

-

Human Capital

+

+

ROI

Complete Training (0) vs. Drop Out (1) +

+

-

Learning Between-Person

+

+

Which Employees Enroll

+

Training Transfer

-

Complete Training Lessons (0) vs. Drop Out (1)

Episodic Learning Within-Person

Personal ROI

+

Enroll in Training

Episodic Training Transfer

Fig. 2. Relationships among training utilization, performance, and financial impact evaluation criteria at the within-person, between-person, and macro levels of analysis. Note: Straight grey arrows demonstrate compositional effects; jagged grey arrows demonstrate compilational effects. Black arrows demonstrate relationships among different effectiveness indicators.

premium placed on employees with marketable skills (Autor, 2014). However, it is important to keep in mind that the value of a college degree may be due more so to employers setting minimum degree requirements as a prerequisite for being hired and signaling that employees have sufficient motivation and cognitive ability to meet the job requirements, rather than college degrees providing directly applicable work-related skills (Caplan, 2017; Spence, 1973). Leadership development is also extremely prevalent and has the potential to generate a substantial personal return on investment. CEOs in the United States earned an average of 11.7 million dollars in 2013, which is 331 times the average worker's salary of $35,293 (Dill, 2014). A variety of factors beyond training—including personality, intelligence, education, and situational factors—play a substantial role in attaining this level of success, but leadership development is invaluable for providing employees with the skillset necessary to rise up the organizational hierarchy (Barling, 2014). The personal ROI from other training and development programs is likely less than the ROI from attaining a college degree or completing leadership development, but the cumulative effects of continuously learning throughout a career should be substantial. Establishing the personal ROI of continuous learning would be invaluable for marketing learning opportunities to employees and increasing the training utilization rate. 5.2. Organizational return on investment Ployhart and Hale (2014) lamented that the human resource literature largely focuses on individuals, ignoring organizational variables. Furthermore, there have been repeated calls in the literature for establishing training's ROI and few studies have addressed this call (Aguinis & Kraiger, 2009). One exception is a meta-analysis that found the relationship between training and objectively assessed financial outcomes (e.g., profit, ROI, return on assets) was 0.04, leading the authors to conclude that “training does not appear to be related to a firm's financial performance” Tharenou, Saks, & Moore, 2007, p. 264). However, offering numerous training programs may enhance an organization's training reputation, increasing the bottom line by attracting better employees and improving current employees' perceptions of the organization. Aligned with his perspective, Tharenou et al. (2007) found training was positively related to managers' and executives' perceptions of financial outcomes (ρ = 0.30). Thus, computing the benefits of training is complicated and it is challenging to place a precise value on the numerous indirect ways that training pays off for organizations. Potentially due to these complications, many organizations do not evaluate ROI, such that 18% of organizations assessed ROI as part of their training evaluation effort in 2009 (Patel, 2010). Although training can be costly, it positively affects organizational profits (Kim & Ployhart, 2014) and measuring ROI is important for making the business case for training. We propose that two conditions must be met for a positive relationship between personal and organizational ROI. Foremost, organizations must reward employees who attain training proficiency to a greater extent than those who fail to attain proficiency. Training programs focusing on critical skills are often delivered to large segments of an organization's workforce, which can result in a substantial ROI for the organization as long as the training benefits outweigh the cost of the course (Morrow, Jarrett, & Rupinski, 1997). The payoff for employees may also be substantial in terms of skill enhancement, but merit raises typically compare an employee to his or her peers, who likely also participated in training. Thus, effective training typically results in a substantial 10

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

Levels of Analysis

Affective Indictors

Macro

Training Reputation

Proximal Pr & Outcomes + -

Enrollment Rate Attrition Rate

+

+

Satisfaction Between-Person

Learning

+

Self-Efficacy

+ -

Motivation

+ +

+

Within-Person

Which Employees Enroll

Episodic Satisfaction

Complete Training (0) vs. Drop Out (1) Training Transfer Decision to Enroll in Training

+

Episodic Learning

+

Episodic SelfEfficacy Episodic Motivation

+ + +

Complete Training Lessons (0) vs. Drop Out (1) Episodic Training Transfer

Fig. 3. Relationships with affective effectiveness indicators at the within-person, between-person, and macro levels of analysis. Note: Jagged grey arrows demonstrate compilational effects. Black arrows demonstrate relationships among different effectiveness indicators. Although not represented in the figure, satisfaction, selfefficacy, and motivation are reciprocally related.

organizational ROI for courses that focus on critical skills but a meager personal ROI, and ROI will only be aligned at the individual and organizational levels of analysis if organizations share the financial gains attained by having a proficiently trained workforce. Although we are unaware of any research that directly examined the percent of organizations that engage in this practice, we presume that a small fraction attain this standard because of the challenges entailed in implementing such a complex system. Second, organizations must integrate their training programs with other human resource functions. Benson, Finegold, and Mohrman (2004) examined the effect of a high-technology manufacturing firm's investment in tuition reimbursement on employee turnover. Tuition reimbursement reduced turnover while employees were enrolled in degree programs. However, turnover increased after employees attained their degrees if they were not subsequently promoted. Thus, personal and organizational ROI will only be related when the most qualified employees are selected for training opportunities (to maximize the organizational ROI) and the employees who use these opportunities to enhance their skillsets are subsequently promoted (to maximize the personal ROI). If learning is not integrated with other human resource functions, employees will only receive a substantial ROI from training if they seek other employment opportunities, causing the organizational ROI to be negative due to the costs of both employee turnover and paying for training. Indeed, investing in training and development often increases employees' performance, while simultaneously increasing job mobility, bargaining power, and turnover (Ployhart et al., in press). ROI can also differ substantially based on whether the program focuses on firm-specific or general knowledge and skills. Becker (1964) proposed that employees are the ultimate beneficiaries of general skills training because they gain knowledge that is valued by a breadth of employers, which can assist them in attaining a raise if they seek other employment opportunities. Thus, firms should focus on specific skills that are unique to their organization's work process so that they maximize their own ROI (Kim & Ployhart, 2014). This suggests that personal and organizational ROI are negatively related if employees leave the firm after completing training. Although personal and organizational ROI capture similar metrics at different levels of analysis, we propose that these evaluation outcomes are unrelated on average, but it is possible to generate a system where they are mutually reinforcing.

6. Relationships among training evaluation criteria Our final objective is to theorize the relationships among training evaluation criteria. Due to the sheer number of evaluation criteria, we utilized two figures to depict how the criteria are related. Specifically, Fig. 2 depicts the relationships among training utilization, performance, and financial impact evaluation criteria, whereas Fig. 3 depicts relationships with affective effectiveness indicators. The vast majority of training evaluations have been conducted at the between-person level of analysis and focused on three criteria—satisfaction, learning, and training transfer. Thus, at the between-person level of analysis, we report the results of empirical investigations and meta-analyses of the literature. However, a scarcity of research has examined the relationships among evaluation criteria at the within-person and macro levels of analysis. Thus, we rely primarily on theoretical insights to predict 11

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

relationships among criteria that have not been subjected to empirical research.

6.1. Effects of training utilization Macro enrollment and attrition rates are expected to predict ROI and human capital (see Fig. 2). It is challenging to attain a positive ROI with low enrollment and/or high attrition because training must have a substantial impact on performance when training expenses are divided across few employees (Cascio & Boudreau, 2011). Moreover, the goal of training is to increase the human capital of an organization's workforce (Kim & Ployhart, 2014). Yet, this goal cannot be realized if employees fail to partake in training or drop out before learning the content. Thus, prior theories of training effectiveness are deficient; failure to include training utilization precludes diagnosing why training may fail to enhance macro effectiveness indicators. At the between-person level of analysis, training utilization enhances personal ROI. Employees cannot benefit from learning opportunities unless if they make time to complete training, suggesting that enrollment has a positive effect on personal ROI while attrition has a negative effect on personal ROI. However, there are a couple of caveats to this rule. Foremost, training must be the best use of employees' time. In most circumstances we presume this condition is met, but employees may attain a more favorable ROI from completing core job duties rather than training when time is limited and those job duties are integral to the success of their firm. Second, the amount of training that was missed and which content was skipped may affect the strength of the attrition/personal ROI relationship, such that skipping material that is already known or irrelevant to one's job may not adversely affect personal ROI. Third, it is important to acknowledge that individuals must be mentally as well as physically present in training to attain a personal ROI. In addition, employees should have greater training transfer if they complete training lessons and the overall program than if they drop out, suggesting that attrition has a negative effect on transfer at both the within- and between-person levels of analysis. If employees are unable to sustain their drive to complete training lessons and the entire program, it is unlikely that they will subsequently muster up the ambition necessary to apply training concepts on the job. This is consistent with the persistence literature, suggesting that motivation transfers across tasks and contexts (Duckworth, Peterson, Matthews, & Kelly, 2007; Grant, 2008). Further, attrition precludes both learning the content presented in the remaining lessons and participating in post-training interventions designed to enhance transfer (e.g., transfer plans, transfer preparation), suggesting that attrition may partially mediate the effect of learning on training transfer.

6.2. Effects of performance effectiveness indicators At the macro level of analysis, human capital has a positive effect on organizational and team performance (Aguinis & Kraiger, 2009). Úbeda Garcia (2005) found human capital development positively affected four firm performance indicators: customer, employee, and shareholder satisfaction as well as sales per employee. Indeed, a recent meta-analysis established that human capital had a moderate effect on organizational performance (ρ = 0.21), especially when human capital focused on firm-specific rather than general knowledge (ρ = 0.30 & 0.17, respectively; Crook et al., 2011). Episodic learning is among the strongest predictors of completing training lessons, such that employees are more likely to drop out following unfavorable than favorable learning (Sitzmann & Johnson, 2012b; Sitzmann et al., 2010). Poor learning suggests that participating in training may be a waste of time and time may be better spent pursuing other goals (Frese & Zapf, 1994; Sitzmann et al., 2010). Employees may elect to drop out rather than investing further time if they are unlikely to attain content mastery, resulting in a strong relationship among these constructs at the within-person level of analysis. Furthermore, a strong between-person relationship is consistent with a meta-analysis of the academic literature demonstrating that students' grade point average has a substantial effect on remaining enrolled in college (ρ = 0.44; Robbins et al., 2004). However, it is important to keep in mind that this research confounds learning and learning performance, so it is unclear whether attrition is driven by failing to understand the content domain or unfavorable performance feedback. Learning should also positively affect personal ROI. This is consistent with research revealing that a one-point rise in high school GPA increases average annual adulthood earnings by 12% for men and 14% for women (French, Homer, Popovici, & Robins, 2014). Moreover, this effect may be partially mediated by attrition because a one-point increase in high school GPA doubles the chance that students will complete a college degree (French et al., 2014). Learning has a moderate between-person effect on training transfer (ρ = 0.24), but the relationship is weaker if transfer assessments are postponed until after employees leave the training environment (ρ = 0.18; Blume et al., 2010). Delayed assessments are aligned with the definition of transfer, so it is important to keep in mind that learning during training does not necessarily translate into substantial changes in job performance. However, between-person effects average across skills that employees mastered and those that they did not completely understand, whereas within-person effects capture the intricacies of this relationship. Thus, we expect that transfer should be greater for skills that employees mastered than those that employees failed to master during training, and this effect size may be stronger than that observed at the between-person level of analysis. Transfer should also positively affect ROI at the macro and between-person levels of analysis. Transfer represents the benefits of training for improving employees' job performance, which serves as the denominator for ROI equations that compare training costs and benefits (Cascio & Boudreau, 2011). However, a high degree of transfer cannot guarantee a macro-level ROI if few employees are trained or training costs are substantial, suggesting that transfer must be considered in conjunction with the enrollment and attrition rates to establish the cost effectiveness of any training initiative. 12

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

6.3. Effects of developing cost effective training The financial impact of training is an essential criterion variable because of its effect on organizational performance. ROI should positively affect organizational performance because investing in employees is a powerful source of competitive advantage that enhances organizational performance both directly and indirectly (e.g., by reducing turnover) (Aguinis & Kraiger, 2009; Barney, 1996; Ichniowski, Shaw, & Prennushi, 1997). However, we are unaware of any research that has empirically tested the training ROI/ organizational performance relationship, and organizational performance is affected by a breadth of policies, procedures, and contextual factors that extend beyond training, which may preclude a strong effect size (Ployhart & Hale, 2014).

6.4. Relationships with affective effectiveness indicators Fig. 3 illustrates the proximal predictors and outcomes of affective effectiveness indicators. At the macro level, the training reputation theoretically affects the enrollment and attrition rates, such that employees will be more likely to enroll and less likely to drop out if training has a favorable reputation (Brown, 2005). This is consistent with a US News and World Report suggesting that moving up in the top 50 university ranking has a positive effect on the number and quality of college applicants (Bowman & Bastedo, 2009). However, we also presume that there are moderators of this relationship (e.g., the reputation will be unrelated to enrollment and attrition in mandatory training), and this relationship may be attenuated because some employees will enroll or drop out without a priori knowledge of training's reputation. At the within- and between-person levels of analysis, pretraining motivation and self-efficacy should positively affect training enrollment. At the between-person level, employees with high motivation and confidence will be more likely to enroll than those low in these affective states. At the within-person level, employees will be more likely to enroll following a surge in self-efficacy or motivation. Both motivation and self-efficacy will be positively affected by mastery experiences on the job and the acquisition of favorable information about training. Thus, it is important to keep in mind that pretraining affect is a product of factors internal (e.g., training reputation) and external (e.g., on the job experiences) to training, so it is not strictly an indicator of training effectiveness. Meta-analytic evidence suggests that satisfaction, motivation, and self-efficacy each have positive relationships with learning at the between-person level of analysis (Colquitt, Conlon, Wesson, Porter, & Ng, 2001). Together, satisfaction, self-efficacy, and motivation overlap with 6% of the variance in post-training declarative knowledge and 25% of the variance in post-training procedural knowledge after controlling for pretraining declarative knowledge (Sitzmann et al., 2008). However, between-person research may have mischaracterized whether learning is a predictor or outcome of affective effectiveness indicators. At the within-person level of analysis, episodic satisfaction does not affect episodic learning, directly contradicting Kirkpatrick's (1996) framework; rather episodic learning predicts episodic satisfaction (Sitzmann et al., 2017). This is consistent with the arguments presented earlier revealing that prior research confounded learning with learning performance, providing an inaccurate assessment of the dynamics of this relationship. Moreover, episodic learning should have a positive effect on episodic self-efficacy, but self-efficacy on average should not account for variance in learning beyond that accounted for by past performance (Sitzmann & Yeo, 2013). Thus, we propose that learning predicts satisfaction and self-efficacy, rather than the converse, at both the within- and between-person levels of analysis. Regarding motivation, we propose a positive and reciprocal relationship with episodic and between-person learning. Motivation increases resource allocation, enhancing learning (Beal et al., 2005; Kanfer & Ackerman, 1989). Attaining high performance, in turn, increases the difficulty of self-set goals, which motivates employees to strive toward goal attainment (Bandura, 1997; Wood & Bandura, 1989). Satisfaction, self-efficacy, and motivation should each negatively affect attrition at both the within- and between-person levels of analysis. Affective states have a direct effect on discretionary behavior (Beal et al., 2005), and satisfaction should enhance task engagement, mitigating the risk of withdrawing from training (Sitzmann et al., 2017). Similar effects should be observed for both selfefficacy and motivation, although empirical support for these propositions is also limited. Employees should be unlikely to abandon their training goals when they are confident in their ability to learn the content domain and have a strong desire to learn. Consistent with this perspective, the probability of dropping out of training increases following declines in self-efficacy and individual differences in motivation and self-efficacy predict goal abandonment (Robbins et al., 2004; Sitzmann, 2012). Finally, the effects of affective effectiveness indicators on training transfer differ across levels of analysis and based on whether satisfaction, motivation, or self-efficacy is the target of the investigation. We are not predicting a direct effect of satisfaction on training transfer because Blume et al.’s (2010) meta-analysis demonstrated that the between-person effect is weak and non-significant (ρ = 0.08). Similarly, it is unreasonable to assume that employees will transfer a particular competency simply because they enjoyed that component of training because satisfaction is affected by a variety of factors that are unrelated to transfer (e.g., entertainment during training; Sitzmann et al., 2008). Post-training self-efficacy and motivation have positive relationships with training transfer at the between-person level of analysis (ρ = 0.22 & 0.29, respectively; Blume et al., 2010). However, we are unaware of any research that has empirically tested these relationships at the within-person level of analysis. Similar to the self-efficacy/performance relationship (Sitzmann & Yeo, 2013; Vancouver et al., 2013), we propose that training transfer will have a more compelling effect on self-efficacy than the converse at both the within- and between-person levels of analysis. Theoretically, motivation should have a positive and reciprocal relationship with training transfer for the same reasons (e.g., resource allocation, goal difficulty) discussed regarding the motivation/learning relationship. 13

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

7. Conclusion We propose a training evaluation framework that addresses the criteria that can be used to assess training effectiveness at multiple levels of analysis. It is our ambition to answer four questions that are imperative for elevating the science underlying training evaluation efforts. Foremost, what outcomes should be assessed to appraise the effectiveness of an organization's overarching training initiative as well as individual training programs? Our response introduces four evaluation taxa. Training utilization captures the extent to which employees take advantage of the instructional curriculum by enrolling in training and completing training programs. Assessing utilization is essential for a comprehensive account of training effectiveness because mediocre usage precludes benefitting from training investments. However, existing models have neglected this evaluation target. Affect encompasses the role of satisfaction, self-efficacy, motivation, and honing a favorable reputation in contributing to the success of training initiatives. Our framework suggests that research has grossly over attended to the between-person level of analysis, despite the fact that affect is a dynamic process and the overarching training reputation is a vital barometer for capturing employees' desire to enroll and exert effort in training. Performance comprises the extent to which employees acquire vital knowledge and skills as a result of training and utilize acquired competencies on the job, ultimately ensuring that the firm has the KSAs necessary to outperform its competitors. Similar to affective effectiveness indicators, researchers have primarily examined between-person differences in performance, despite the fact that correcting performance deficiencies necessitates a dynamic approach and improvements in human capital and organizational performance are the ultimate objective of any training initiative. At last, the financial impact reveals whether training is a sound investment for the firm and spotlights ROI as an imperative comparison of training costs relative to the benefits. ROI has been a core component of evaluation models since the 1950s (Kirkpatrick, 1959), but the knowledge that has accrued in the last 58 years is underwhelming. Second, what is the appropriate level of analysis for assessing training evaluation criteria? Our framework reveals six criteria that should be assessed at the macro level of analysis, eight criteria that should be assessed at the between-person level of analysis, and seven criteria that should be assessed at the within-person level of analysis. Together these 21 criteria provide a comprehensive account of the strengths and weaknesses of training initiatives. In response to this second inquiry, we highlighted several crucial discrepancies between theoretical conceptualization and empirical operationalization of training evaluation criteria as well as how these discrepancies have led to imprudent evaluation practices. For example, self-efficacy is inherently a within-person process—it dynamically evolves as employees gain experience in a domain and plays a crucial role in self-assessments of the level of effort that must be exerted to attain work-related goals. Yet, the vast majority of self-efficacy research has focused on the between-person level of analysis (Sitzmann & Yeo, 2013). Another example is that satisfaction is the most frequently assessed evaluation criterion, despite the fact that the majority of training objectives target performance effectiveness indicators (Sitzmann et al., 2017). It is an irrefutable fact that people behave in a manner to attain favorable performance evaluations (Kerr, 1975; Sitzmann et al., 2017). Thus, evaluating training often exclusively with satisfaction surveys epitomizes the folly of hoping for competency development while rewarding instructors for making students happy. Further, satisfaction surveys are routinely administered post-training, but dissatisfaction with a single element of training may have an undue influence on this overarching evaluation measure. If the evaluation team deems it wise to make subsequent offerings more satisfying, focusing on episodic satisfaction is essential for detecting the pleasant and unpleasant aspects of training. The final discrepancy that we wish to highlight is the fact that macro evaluation metrics are of preeminent concern to firms' executives, but a scarcity of research has examined evaluation criteria at this level of analysis. Bridging the divide between micro and macro training evaluation criteria is imperative for demonstrating the role of training in advancing a firm's competitive advantage and ensuring continued investment in this human resource function. The last two questions address how constructs are related both across levels of analysis and to other evaluation criteria. Specifically, when is it appropriate to aggregate responses from a lower level of analysis to assess training effectiveness at a higher level of analysis? And how are training evaluation criteria interrelated? It is unlikely that any evaluation endeavor is going to examine all 4 taxa nor all 21 criteria. Rather, we concur with Kraiger (2002) that the first step of any evaluation effort involves questioning the purpose of the evaluation. For example, is the goal to detect changes in learners, whether the overarching organization is benefiting from training, precisely where training improvements are needed, or another purpose? Once the purpose is determined, the training evaluation effort should begin by addressing the construct that is directly aligned with that objective. As an illustration, if the purpose is to determine which employees attained proficiency by the end of training, between-person learning assessments should be administered to compare post-training content mastery across employees. Yet, if the purpose is to determine why employees are not proficient in trained skills by the end of the course, learning should be gauged at the within-person level to detect when learning began to decline. After evaluating a single outcome, this framework can be used to determine whether other benefits may also have been derived from training. Understanding the interrelatedness of evaluation criteria is essential for gauging the depth of knowledge that can be inferred by any evaluation effort. The figures advanced by our theoretical framework illustrate how evaluation criteria are related and whether there is a compositional or compilational relationship among criteria across levels of analysis. An examination of the figures suggests that training effectiveness is primarily a bottom-up process. Organizational theory often neglects bottom-up processes, and the focus is typically on compositional processes whenever bottom-up processes are addressed, which is a limitation of current multilevel theories (Kozlowski & Klein, 2000). The goal of organizational science is to understand organizations and how employees in aggregate contribute to macro phenomena so proposing bottom-up compositional and compilational processes is another contribution of the current framework. Maintaining a strict policy whereby evaluation criteria are only aggregated to higher levels of analysis when the relationships are compositional is essential for avoiding the cross-level fallacy. Ultimately, we hope that the 14

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

current framework facilitates intellectual progress on the interplay among evaluation metrics both within and across levels of analysis. References Aguinis, H. (2009). Performance management. NJ: Pearson Prentice Hall. Aguinis, H., & Kraiger, K. (2009). Benefits of training and development for individuals and teams, organizations, and society. Annual Review of Psychology, 60, 451–474. Ajzen, I. (1985). From intentions to actions: A theory of planned behavior. In J. Kuhl, & J. Beckman (Eds.), Action-control: From cognition to behavior (pp. 11–39). Heidelberg: Springer. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. Alliger, G. M., & Janak, E. A. (1989). Kirkpatrick's levels of training criteria: Thirty years later. Personnel Psychology, 42(2), 331–342. Alliger, G. M., Tannenbaum, S. I., Bennett, W., Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50, 341–358. ASTD (2013). 2013 State of the industry. Alexandria, VA: American Society for Training & Development. Autor, D. H. (2014). Skills, education, and the rise of earnings inequality among the “other 99 percent”. Science, 344, 843–851. Baldwin, T. T., & Ford, J. K. (1988). Transfer in training: A review and directions for future research. Personnel Psychology, 41, 63–105. http://dx.doi.org/10.1111/j. 1744-6570.1988.tb00632.x. Baldwin, T. T., Ford, J. K., & Blume, B. D. (2009). Transfer of training 1988-2008: An updated review and agenda for future research. International Review of Industrial and Organizational Psychology, 24, 41–70. Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191–215. Bandura, A. (1989). Human agency in social cognitive theory. American Psychologist, 44, 1175–1184. Bandura, A. (1997). Self-efficacy: The exercise of control. New York: Freeman. Barling, J. (2014). The science of leadership: Lessons from research for organizational leaders. USA: Oxford University Press. Barney, J. (1996). The resource-based theory of the firm. Organization Science, 7(5), 469. Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341–347. Beal, D. J., Weiss, H. M., Barros, E., & MacDermid, S. M. (2005). An episodic process model of affective influences on performance. Journal of Applied Psychology, 90(6), 1054–1068. Becker, G. S. (1964). Human capital: A theoretical and empirical analysis with special reference to education. Chicago: University of Chicago Press. Bell, B. S., & Kozlowski, S. W. J. (2002). Adaptive guidance: Enhancing self-regulation, knowledge, and performance in technology-based training. Personnel Psychology, 55, 267–306. Benson, G. S., Finegold, D., & Mohrman, S. A. (2004). You paid for the skills, now keep them: Tuition reimbursement and voluntary turnover. Academy of Management Journal, 47(3), 315–331. Bilginsoy, C. (2003). The hazards of training: Attrition and retention in construction industry apprenticeship programs. Industrial and Labor Relations Review, 57, 54–67. Blume, B., Ford, J., Baldwin, T., & Huang, J. (2010). Transfer of training: A meta-analytic view. Journal of Management, 36(4), 1065–1105. Bowman, N. A., & Bastedo, M. N. (2009). Getting on the front page: Organizational reputation, status signals, and the impact of US News and World Report on student decisions. Research in Higher Education, 50(5), 415–436. Brown, K. G. (2005). An examination of the structure and nomological network of trainee reactions: A closer look at “smile sheets”. Journal of Applied Psychology, 90(5), 991. Caplan, B. (2017). The case against education. New Jersey: Princeton University Press. Carver, C. S., & Scheier, M. F. (1998). On the self-regulation of behavior. New York: Cambridge University Press. Cascio, W. F., & Boudreau, J. W. (2011). Investing in people: The financial impact of human resource initiatives(2nd ed.). Upper Saddle River, NJ: Pearson Education. Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. Colquitt, J. A., Conlon, D. E., Wesson, M. J., Porter, C. O. L. H., & Ng, K. Y. (2001). Justice at the millennium: A meta-analytic review of 25 years of organizational justice research. Journal of Applied Psychology, 86, 425–445. Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85(3), 678–707. Crook, T. R., Todd, S. Y., Combs, J. G., Woehr, D. J., & Ketchen, D. J. (2011). Does human capital matter? A meta-analysis of the relationship between human capital and firm performance. Journal of Applied Psychology, 96(3), 443–456. Dalton, D. R., Krackhardt, D. M., & Porter, L. W. (1981). Functional turnover: An empirical assessment. Journal of Applied Psychology, 66(6), 716–721. Delaney, J. T., & Huselid, M. A. (1996). The impact of human resource management practices on perceptions of organizational performance. Academy of Management Journal, 39(4), 949–969. Detterman, D. K. (1993). The case for the prosecution: Transfer as an epiphenomenon. Westport, CT: Ablex Publishing. Dill, K. (2014). Report: CEOs Earn 331 Times As Much As Average Workers, 774 Times As Much As Minimum Wage Earners. Retrieved from http://www.forbes.com/sites/ kathryndill/2014/04/15/report-ceos-earn-331-times-as-much-as-average-workers-774-times-as-much-as-minimum-wage-earners/. DiMaggio, P., & Powell, W. W. (1983). The iron cage revisited: Collective rationality and institutional isomorphism in organizational fields. American Sociological Review, 48(2), 147–160. Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087. Facteau, J. D., Dobbins, G. H., Russell, J. E. A., Ladd, R. T., & Kudisch, J. D. (1995). The influence of general perceptions of the training environment on pretraining motivation and perceived training transfer. Journal of Management, 21, 1–25. Ford, J. K., Yelon, S. L., & Billington, A. Q. (2011). How much is transferred from training to the job? The 10% delusion as a catalyst for thinking about transfer. Performance Improvement Quarterly, 24(2), 7–24. Fredrickson, B. L., & Kahneman, D. (1993). Duration neglect in retrospective evaluations of affective episodes. Journal of Personality and Social Psychology, 65(1), 45. French, M. T., Homer, J. F., Popovici, I., & Robins, P. K. (2014). What you do in high school matters: High school GPA, educational attainment, and labor market earnings as a young adult. Eastern Economic Journal. Frese, M., & Zapf, D. (1994). Action as the core of work psychology: A German approach. In M. Dunnette, L. Hough, & H. C. Triandis (Vol. Eds.), Handbook of industrial and organizational psychology. vol. 4 (2), (pp. 271–340). Palo Alto, CA: Consulting Psychologists Press. Georgensen, D. L. (1982). The problem of transfer calls for partnership. Training and Development Journal, 36, 75–78. Gollwitzer, P. M. (1999). Implementation intentions: Strong effects of simple plans. American Psychologist, 54(7), 493. Grant, A. M. (2008). Does intrinsic motivation fuel the prosocial fire? Motivational synergy in predicting persistence, performance, and productivity. Journal of Applied Psychology, 93(1), 48. Hatch, N. W., & Dyer, J. H. (2004). Human capital and learning as a source of sustainable competitive advantage. Strategic Management Journal, 25(12), 1155–1178. Holton, E. F., III (1996). The flawed four-level evaluation model. Human Resource Development Quarterly, 7, 5–21. Holton, E. F., III, & Baldwin, T. T. (2003). Making transfer happen: An action perspective on learning transfer systems. In E. F. Holton, & T. T. Baldwin (Eds.), Improving learning transfer in organizations (pp. 3–15). San Francisco: Jossey-Bass. Hurtz, G. M., & Williams, K. J. (2009). Attitudinal and motivational antecedents of participation in voluntary employee development activities. Journal of Applied Psychology, 94, 635–653. Ichniowski, C., Shaw, K., & Prennushi, G. (1997). The effects of human resource practices on manufacturing performance: A study of steel finishing lines. American

15

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

Economic Review, 87(3), 291–313. Katz, M. L., & Shapiro, C. (1987). R and D rivalry with licensing or imitation. The American Economic Review, 402–420. Kanfer, R., & Ackerman, P. L. (1989). Motivation and cognitive abilities: An integrative aptitude-treatment interaction approach to skill acquisition. Journal of Applied Psychology, 74, 657–690. Kennedy, M. T., & Fiss, P. C. (2009). Institutionalization, framing, and diffusion: The logic of TQM adoption and implementation decisions among US hospitals. Academy of Management Journal, 52(5), 897–918. Kerr, S. (1975). On the folly of rewarding A, while hoping for B. Academy of Management Journal, 18(4), 769–783. Kim, Y., & Ployhart, R. (2014). The effects of staffing and training on firm productivity and profit growth before, during, and after the great recession. Journal of Applied Psychology, 99(3), 361–389. Kirkpatrick, D. L. (1959). Techniques for evaluating training programs. Journal of American Society of Training Directors, 13(3), 21–26. Kirkpatrick, D. L. (1996). Evaluation. In R. L. Craig (Vol. Ed.), The ASTD training and development handbook. Vol. 4, (pp. 294–312). New York: McGraw-Hill. Korn, M., & Levitz, J. (2013). Online courses look for a business model. The Wall Street Journal. http://online.wsj.com/news/articles/ SB10001424127887324339204578173421673664106 2 January. Kozlowski, S. W. J., Brown, K. G., Weissbein, D. A., Cannon-Bowers, J. A., & Salas, E. (2000). A multi-level perspective on training effectiveness: Enhancing horizontal and vertical transfer. In K. J. Klein, & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations (pp. 157–210). San Francisco: Jossey-Bass. Kozlowski, S. W. J., & Klein, K. J. (2000). A multilevel approach to theory and research in organizations: Contextual, temporal, and emergent processes. In K. Klein, & S. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 3–90). San Francisco, CA: Jossey-Bass. Kozlowski, S. W., & Salas, E. (1997). A multilevel organizational systems approach for the implementation and transfer of training. Improving training effectiveness in work organizations, 247, 287. Kraiger, K. (2002). Decision-based evaluation. In K. Kraiger (Ed.), Creating, implementing, and maintaining effective training and development: State-of-the-art lessons for practice (pp. 331–375). Mahwah, NJ: Jossey-Bass. Kraiger, K., & Ford, J. K. (2007). The expanding role of workplace training: Themes and trends influencing training research and practice. Historical perspectives in industrial and organizational psychology (pp. 281–309). Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328. http://dx.doi.org/10.1037/0021-9010.78.2.311. Lange, D., Lee, P. M., & Dai, Y. (2011). Organizational reputation: A review. Journal of Management, 37, 153–184. Lavelle, L. (2012). College ROI: What we found. Bloomberg Business. Retrieved from http://www.bloomberg.com/bw/articles/2012-04-09/college-roi-what-we-found. Mathieu, J. E., & Tesluk, P. E. (2010). A multilevel perspective on training and development effectiveness. In S. Kozlowski, & E. Salas (Eds.), Learning, training, and development in organizations (pp. 405–440). New York: Routledge. Morrow, C. C., Jarrett, M. Q., & Rupinski, M. T. (1997). An investigation of the effect and economic utility of corporate-wide training. Personnel Psychology, 50(1), 91–117. Noe, R. A. (1986). Trainees' attributes and attitudes: Neglected influences on training effectiveness. Academy of Management Review, 11, 736–749. Noe, R. A. (2008). Employee training and development(4th ed.). Boston, MA: McGraw Hill. Noe, R. A., Clarke, A. D., & Klein, H. J. (2014). Learning in the twenty-first-century workplace. Annual Review of Organizational Psychology and Organizational Behavior, 1, 4.1–4.31. Patel, L. (2010). 2010 state of the industry: ASTD's definitive review of workplace learning and development trends. Alexandria, VA: ASTD. Phillips, J. J., & Phillips, P. (2007). Measuring return on investment in leadership development. In K. M. Hannum, J. W. Martineau, & C. Reinalt (Eds.), The handbook of leadership development evaluation (pp. 137–166). Hoboken, NJ: John Wiley and Sons. Ployhart, R. E., Call, M. L., & McFarland, L. A. (in press). Learning, human capital, and organizational value. In R. Noe. & J. Ellingson (Eds.), Autonomous learning in the workplace. Taylor & Francis. Ployhart, R. E., & Hale, D. (2014). The fascinating psychological microfoundations of strategy and competitive advantage. Annual Review of Organizational Psychology and Organizational Behavior, 1, 14.1–14.28. Ployhart, R. E., & Moliterno, T. P. (2011). Emergence of the human capital resource: A multilevel model. Academy of Management Review, 36(1), 127–150. Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261–288. Salas, E., DiazGranados, D., Klein, C., Burke, C. S., Stagl, K. C., Goodwin, G. F., & Halpin, S. M. (2008). Does team training improve team performance? A meta-analysis. Human Factors, 50(6), 903–933. Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-Jentsch, K. A. (2012). The science of training and development in organizations: What matters in practice. Psychological Science in the Public Interest, 13(2), 74–101. Schmidt, A. M., & DeShon, R. P. (2007). What to do? The effects of discrepancies, incentives, and time on dynamic goal prioritization. Journal of Applied Psychology, 92(4), 928–941. Simons, T., & Roberson, Q. (2003). Why managers should care about fairness: The effects of aggregate justice perceptions on organizational outcomes. Journal of Applied Psychology, 88(3), 432–443. Sitzmann, T. (2012). A theoretical model and analysis of the effect of self-regulation on attrition from voluntary online training. Learning and Individual Differences, 22(1), 46–54. Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied Psychology, 93, 280–295. Sitzmann, T., Brown, K. G., Ely, K., Kraiger, K., & Wisher, R. A. (2009). A cyclical model of motivational constructs in web-based courses. Military Psychology, 21(4), 534–551. Sitzmann, T., & Ely, K. (2010). Sometimes you need a reminder: The effects of prompting self-regulation on regulatory processes, learning, and attrition. Journal of Applied Psychology, 95(1), 132–144. Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137, 421–442. Sitzmann, T., Ely, K., Bell, B. S., & Bauer, K. N. (2010). The effects of technical difficulties on learning and attrition during online training. Journal of Experimental Psychology Applied, 16, 281–292. Sitzmann, T., & Johnson, S. K. (2012a). The best laid plans: Examining the conditions under which a planning intervention improves learning and reduces attrition. Journal of Applied Psychology, 97(5), 967. Sitzmann, T., & Johnson, S. K. (2012b). When is ignorance bliss? The effects of inaccurate self-assessments of knowledge on learning and attrition. Organizational Behavior and Human Decision Processes, 117, 192–207. Sitzmann, T., & Johnson, S. (2014). The paradox of seduction by irrelevant details: How irrelevant information helps and hinders self-regulated learning. Learning and Individual Differences, 34, 1–11. Sitzmann, T., Song, Y., & Wang, M. (2017). The folly of hoping for achievement while rewarding customer satisfaction: Reevaluating the relationship between course satisfaction and learning. Unpublished manuscript. Business School, University of Colorado Denver. Sitzmann, T. & Weinhardt, J. M. (in press). Training engagement theory: A multilevel perspective on the effectiveness of work-related training. Journal of Management. Sitzmann, T., & Yeo, G. (2013). A meta-analytic investigation of the within-person self-efficacy domain: Is self-efficacy a product of past performance or a driver of future performance? Personnel Psychology, 66, 531–568. Spence, M. (1973). Job market signaling. The Quarterly Journal of Economics, 355–374. Strang, D., & Macy, M. W. (2001). In Search of Excellence: Fads, Success Stories, and Adaptive Emulation 1. American Journal of Sociology, 107(1), 147–182. Strang, D., & Meyer, J. W. (1993). Institutional conditions for diffusion. Theory and Society, 22(4), 487–511.

16

Human Resource Management Review xxx (xxxx) xxx–xxx

T. Sitzmann, J.M. Weinhardt

Strang, D., & Soule, S. A. (1998). Diffusion in organizations and social movements: From hybrid corn to poison pills. Annual Review of Sociology, 24(1), 265–290. Sun, S., Vancouver, J. B., & Weinhardt, J. M. (2014). Goal choices and planning: Distinct expectancy and value effects in two goal processes. Organizational Behavior and Human Decision Processes, 125(2), 220–233. Switzer, K. C., Nagy, M. S., & Mullins, M. E. (2005). The influence of training reputation, managerial support, and self-efficacy on pre-training motivation and perceived training transfer. Applied HRM Research, 10(1), 21–34. Tannenbaum, S. I., Mathieu, J. E., Salas, E., & Cannon-Bowers, J. A. (1991). Meeting trainees' expectations: The influence of training fulfillment on the development of commitment, self-efficacy, and motivation. Journal of Applied Psychology, 76, 759–769. Tharenou, P., Saks, A. M., & Moore, C. (2007). A review and critique of research on training and organizational-level outcomes. Human Resource Management Review, 17(3), 251–273. Uttl, B., White, C. A., & Gonzalez, D. W. (2016). Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation. Vancouver, J. B. (2005). The depth of history and explanation as benefit and bane for psychological control theories. Journal of Applied Psychology, 90(1), 38–52. Vancouver, J. B., & Kendall, L. N. (2006). When self-efficacy negatively relates to motivation and performance in a learning context. Journal of Applied Psychology, 91, 1146–1153. Vancouver, J. B., More, K. M., & Yoder, R. J. (2008). Self-efficacy and resource allocation: Support for a nonmonotonic, discontinuous model. Journal of Applied Psychology, 93, 35–47. Vancouver, J. B., Weinhardt, J. M., Warren, M., Covey, A., Purl, J., Milakovic, A., & Li, X. (2013). Do management scholars mistakenly believe in the capacity of selfefficacy? In D. Svyantek, & K. Mahoney (Eds.), Received wisdom, kernels of truth, and boundary conditions in organizational studies (pp. 77–103). Charlotte, NC: Information Age Publishing. Weiss, H. (1990). Learning theory and industrial and organizational psychology. In M. D. Dunnette, & L. M. Hough (Vol. Eds.), Handbook of industrial and organizational psychology. Vol. 1, (pp. 171–222). Palo Alto, CA: Consulting Psychologists Press. Wood, R. E., & Bandura, A. (1989). Impact of conceptions of ability on self-regulatory mechanisms and complex decision making. Journal of Personality and Social Psychology, 56, 407–415.

17