Innovations in job analysis: Development and application of metrics to analyze job data

Human Resource Management Review 16 (2006) 310 – 323 www.socscinet.com/bam/humres

Innovations in job analysis: Development and application of metrics to analyze job data Lauren E. McEntire, Lesley R. Dailey, Holly K. Osburn, Michael D. Mumford ⁎ Department of Psychology, University of Oklahoma, 455 W. Lindsey, Room 705, Norman, OK 73019, United States

Abstract Job analysis is an integral part of any human resource function. Recent advancements in technology and changing worker environments have drastically altered the means by which job analysis data are collected and stored. These changes have led to an increase in the amount of data that is collected and the potential for the data to inform complex decision making. However, due to a lack of tools available for configuring and analyzing data, human resource professionals are often unable to keep themselves abreast of changes in their workforce, make complex decisions using job data, and facilitate communication across jobs, job families or departments in their organization. As a result, advanced methods for analysis of job data are needed. Metrics are quantitative algorithms applied to job data that aid in decision making in areas such as recruitment, selection, transferability, promotion, training, and development. Metrics are a sophisticated, user-friendly approach to analyzing job data that have the potential to meet the needs of human resource professionals in today's dynamic workplace. The development of metrics, their application and benefit to human resource professionals, and their use of O⁎NET are discussed. © 2006 Elsevier Inc. All rights reserved. Keywords: Job analysis; Metrics; Job data

Job analysis involves the collection of various types of job data and worker requirements in an organization (Harvey, 1991; McCormick, 1976). The information obtained from a job analysis serves as a foundation for most, if not all, human resource (HR) related activities and issues (Dessler, 2003; Jeanneret & Strong, 2003; Riggio, 1990). In fact, job analysis is important to an organization because it serves as a strategic human resource management (HRM) practice through which organizations gain a better understanding of their jobs and their workers (Schneider & Konz, 1989; Siddique, 2004). There are a number of ways that job analysis information can be applied to HRM functions (Harvey, 1991). In its most generic form, job analysis data provides a point of reference to the user (i.e., human resource professional, job incumbent or manager). However, the potential of job analysis data to aid in organizational human resource functions extends far beyond the scope of its current use (Sanchez, 1994; Walker, 1990). There are a vast range of possibilities for application and use in human resource decision making if the methods used for analyzing job analysis data can be refined and updated. ⁎ Corresponding author. Tel.: +1 405 325 5583; fax: +1 405 325 4737. E-mail address: [email protected] (M.D. Mumford). 1053-4822/$ - see front matter © 2006 Elsevier Inc. All rights reserved. doi:10.1016/j.hrmr.2006.05.004

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

311

Human resource decisions using job analysis data often rely on loose inferences or qualitative hunches rather than using the data to its full potential (i.e., in dynamic and complex ways to draw objective conclusions) (Chermack, 2003; Korte, 2003; Murphy & Zandvakili, 2000; Sanchez & Levine, 2000). This tendency to ineffectively apply and make decisions using job analysis data are often due to the unavailability of simple means and tools for data configuration and analysis (Harvey, 1991). The issues resulting from this deficiency in tools for human resource professionals are three-fold. First, human resource professionals are not able to keep their job analysis outcomes current with respect to the status of their human capital, changes in organizational needs, and advancements in technology and equipment. In other words, decisions made using the data are not a reflection of the current state of their workforce because it is not analyzed and applied quicker than job changes are occurring. Second, job analysis data are highly underutilized especially in the context of complex decision making issues. Job analysis is often only used for simple, routine decision making or administrative reports. Third, job analysis decisions are not portable across jobs, job families, departments or organizations. Inferences and decisions are often only applicable to one job or a set of individuals and are not flexible for further interpretation. Therefore, the purpose of this article is to present a method for configuring and analyzing large amounts of job analysis data that can address these issues and ultimately aid in human resource decision making. This method provides a quantitative approach for deriving value from job analysis data that is not only less cumbersome than traditional methods, but also holds the potential to reduce the monetary costs that are often associated with job analysis. In addition and prior to the presentation of metrics, we would like to discuss the three aforementioned issues related to the use of job analysis data in more detail as well as the utility of the Occupational Information Network (O⁎NET) when collecting and analyzing job data. Issue 1: Workforce and technological trends affecting job analysis Today's professional and industrial work settings have evolved in recent years to become fast paced, dynamic environments filled with technology and innovation (Cascio, 1995; Walker, 1990). Work environments and worker requirements have transformed drastically in the last few decades to keep up with these trends and they are expected to continue in this direction (McCann, 2004; Schneider & Konz, 1989; Walker, 1999). Some contributing factors to this work evolution are movements toward broad and cross-functional job responsibilities (Cunningham, 1996), dissolving manager–labor distinctions between professional workers, regular shifts in work responsibilities, global competition, free trade, and prevalence of teamwork and self-managing teams (Sanchez, 1994). Additionally, organizations are using automation in virtually every administrative and human resource activity to increase speed of performance, garner more information, and reduce human error (Buckley, Minette, Joy, & Michaels, 2004; McDonald & Cornetto, 2005). Recent technological advancements in the methods for collection and storage of job data have embraced these changes in the nature of work. (Crespin & Austin, 2002). Computer applications and software are now prevailing methods for collecting job analysis data. In fact, many organizations have completely eliminated the use of interviews and paper–pencil methods (Patterson & Lindsey, 2003; Sanchez, 1994). Computerized methods for collecting job analysis data are often transferred to an online, interactive format where all data are collected on the computer using the internet. As a result of these technological advances, job analysis data can be collected, stored, maintained, and updated with relative ease. Though the dynamic changes in the workforce can be documented with the technology available for job data collection, if the tools needed for human resource professionals to analyze the data are not present or utilized then the meaningful results and decision making opportunities may be forfeited. It is possible that the job information acquired using more time consuming, traditional methods of analysis will be obsolete because the job data are already in need of an update. Issue 2: Underutilization of job analysis data for decision making While there are clear advantages to collecting job analysis data using computers and the internet, this method has created new issues for human resource practitioners to address. One issue that was not applicable when paper-pencil methods for collecting job data were in place is the sheer amount of data that can be collected (Patterson & Lindsey, 2003). Online formats allow for a much larger dataset than has been possible in the past. Because of this, human resource professionals must be quite specific in the ways they intend to use the data a priori. This specificity allows for the creation

312

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

of interrelationships among the data during the collection process, thereby allowing it to be organized in meaningful ways for the user. Otherwise, it is possible to obtain a dataset that is so large and unorganized that it is difficult to analyze. Even with an organized data collection and linkage process, databases used to store job analysis data are typically very large. These large databases prove difficult for users to efficiently apply the data to its greatest capability, especially when users lack the necessary skills and procedural steps needed to conduct data analysis. Consequently, analysis of job data has become increasingly time consuming, disorganized and ineffective. The significant amount of data makes commonly used qualitative approaches difficult to apply effectively. It is conceivable that the decisions made using qualitative analyses may lack the complexity and influence necessary to justify the resources spent on the analysis. Though we do not discount the use of qualitative methods, other options for effectively analyzing workforce data to make complex decisions using quantitative methods are needed. Issue 3: Job analysis data portability and usefulness of O⁎NET Human resource professionals often lack the resources in their job data and data analysis techniques to combine data and make decisions at multiple levels. Job analysis data has the potential to be analyzed so that information can be acquired about several jobs or job families and consequently multi-level decisions can be made. However, the data must be structured to accommodate such an analysis. The job data must have the capability for aggregation. This is especially true when a job analysis is needed across several jobs and job families as is the case in many large organizations including government and military. Without a method to aggregate the data, decisions made using the data across levels will not be valid (Harvey & Wilson, 2000). For example, a human resource professional should not quantitatively compare the skills used on two different jobs if the skills for each job are not from the same skill taxonomy. Occupational specific skills cannot be compared without a structured link between the two skillsets; therefore inferences cannot be made about the two jobs in terms of skills (e.g., similarity, complexity). A solution to this problem of aggregation can be attained through the use of an online database called O⁎NET. A significant part of the development and application of metrics uses standardized data from O⁎NET. The Occupational Information Network replaced the Dictionary of Occupational Titles (U.S. Department of Labor, 1991) as a standardized, comprehensive, and online system available for performing worker and job oriented job analysis and developing descriptions of jobs. This system can be used as a starting point to describe most any job in the organizations of today and in the future (Mariani, 1999, 2001). The system is designed to present job information at both a general and an occupation-specific level (Peterson, Mumford, Borman, Jeanneret, & Fleishman, 1999; Peterson et al., 2001). There are three key advantages to using O⁎NET based job analysis data in the development of data analysis techniques. The first advantage is its use of multiple windows into the world of work. The “multiple windows” approach considers the fact that jobs can be described and understood from a variety of different perspectives (Mumford & Peterson, 1999). That is, one approach might be to describe a job in terms of the abilities necessary to do the work, while another approach might be to describe the job in terms of the context in which the job occurs. The second advantage of using O⁎NET as a framework for the development of data analysis techniques is its use of cross-job descriptors which provide a common language to describe and compare jobs. In other words, these descriptors portray the job in terms of broader variables that cut across occupations. They cut across occupations because O⁎NET skills, abilities, Generalized Work Activities (GWAs), and knowledges are standardized taxonomies that relate to all types of jobs. This particular feature makes the O⁎NET system highly efficient because new descriptive systems do not have to be developed for each new job. The third advantage of using O⁎NET is its use of a hierarchical organization of the variables used as descriptor domains. In other words, the O⁎NET structure includes more general descriptors that can be seen as subsuming the more specific descriptors (Mumford & Peterson, 1999). This hierarchical structure is central to the development of data analysis techniques because it allows for the combination of data. Though each element (i.e., skills, abilities, GWAs, knowledges) is unique, all elements can be aggregated for analysis because they come from standardized taxonomies and are meaningful when related to each other. For example, users can combine broad descriptors, like GWAs with more specific descriptors, like basic and cross-functional skills. Though these elements are different in the level of their description, they are both still standardized elements of the job and hence can be compared with other related jobs (as related jobs will have these same elements). This means of aggregation expands the capabilities previously available in analyzing job data.

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

313

In sum, O⁎NET uses standardized work activities and skill descriptions to provide a common language to describe and characterize various jobs and occupations (Borman, 1996). The framework provided by O⁎NET is highly efficient because it provides a common point of reference when describing and comparing jobs, and thus eliminates the need to develop new descriptive systems when describing new jobs (Peterson et al., 2001). In addition, this standardized framework makes it possible to aggregate data not only within a job, but also between jobs in an occupational family. By providing a way to aggregate data, the O⁎NET framework allows for the creation of data management formulas. These formulas, often called metrics, can be used with job analysis data to answer questions about the workforce. The introduction of O⁎NET and various technological advances have led to the possibility for increased number of applications for job analysis data. Subsequently, human resource professionals are in need of a means to analyze data that is progressive enough to address dynamic and demanding work environments and advancements in technology as well as allow portable decisions across levels, yet simple enough to be used every day to help make decisions with ease. The remainder of this article will discuss a method for using data collected in the job analysis process for human resource decision making. In particular, metrics are described in terms of their development, future validation, and their benefit to numerous areas of HRM. 4. Metrics Metrics use quantitative and qualitative job analysis data and apply step-by-step mathematical formulas, or algorithms, to answer workforce analysis questions. These step-by-step formulas allow for the transformation of job analysis data into more manageable and easily interpretable components, thus providing a more economical and effective way of using job analysis data. The use of metrics can mesh data analysis and decision making to aid those making personnel related decisions applicable to the workforce, job, or employee of interest. For example, metrics can be employed that will help the organization determine the most qualified individuals for promotion or to help determine the skills that an employee needs to be successful on the job. The nature of the algorithms employed and the type of data used in metrics allow for more objectivity in answering workforce questions, which ultimately leads to more reliable conclusions. To ensure reliability and proper aggregation of data, metrics use O⁎NET and typical organizational job analysis information such as worker and job oriented data. O⁎NET elements are aggregated together, weighted accordingly, and analyzed together to draw the most meaningful conclusion. The metrics discussed in this article are the result of a collaborative research effort in a contract with the Department of Defense, SkillsNET® Corporation and the University of Oklahoma. The goal of this effort was to produce a userfriendly, innovative and quantitative means to address a variety of HR issues in the Navy. Three specific problem areas were identified as the primary focus points for the development of metrics. These areas included 1) metrics to organize and analyze basic job data, 2) metrics to aid in recruitment, selection, and transferability related decisions and 3) metrics to aid in training related decisions. Several metrics were developed to address each area in as many meaningful ways as possible. Additional metrics were developed that addressed areas such as using complex job data and military mission data. All elements of the most recent Navy job analysis, O⁎NET, and some external data sources were used in the metric development and testing process. All the metrics were developed, tested, and will be validated by trained Industrial/Organizational psychologists and statisticians. No metric has received thorough validation at this stage in the development effort. The next section describes the metric development process, followed by a detailed illustration. For a full list of developed metrics see Appendix A. 4.1. Metric development process The development of any metric involves a systematic process that consists of a series of key steps. In this section an overview of the steps involved in metric development will be presented. This overview will provide a general view of the issues one must consider and address when creating a metric. To provide more insight into the development process two examples are provided in the next section. These examples are provided in a detailed manner to illustrate the development of a metric from conception to current version. Additionally, Fig. 1 may be referenced as a means to visually represent the metric development process prior to validation.

314

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

Fig. 1. Metric development process.

The first step in metric development is to determine what workforce analysis issues are of interest to the organization. Typically these questions are developed by human resource professionals in an organization through discussion of the areas in which an analysis of job data would create the most benefit. Once the question(s) of interest

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

315

are generated, a rationale for the importance of each question to the organization is developed. This rationale helps to clarify the workforce analysis question and to identify what specific, measurable benefit will arise from answering each question. Components of the rationale include how the information will be used in an organizational setting, how it will contribute to management strategy, and the added value this particular workforce information could bring to the organization. This identification of workforce questions and the impact of the questions on the organization can initiate a more in depth analysis of the area of interest. This analysis allows the organization to more effectively assess and target the areas that could benefit from additional examination and can provide the foundation for the next step in the process, the development of algorithm components to be used in the metric. There are three steps involved in determining the components of the metric algorithm. First, a list of key elements is developed. These key elements are broad in scope and are essentially the basic ingredients of the metric. The key elements represent components that will contribute to answering the workforce analysis question. Second, the list of key elements is shortened so that only the elements deemed most relevant to the workforce analysis question are included. Third, a list of potential sources is identified. A source is a piece of information or set of data that can populate a key element. For example, it may be determined that a key element is a list of abilities, and source options for populating this key element are O⁎Net abilities, occupation specific abilities, or technical abilities. Sources are identified by researching available data options relevant to the key element. Once a list of potential sources has been developed, an examination of these sources is needed to establish which will be kept and which will be eliminated. At this point, users ascertain what specific pieces of data are most appropriate to be used as a source. This is typically obtained from job analysis information. For example, users would determine whether an existing taxonomy such as O⁎NET is available that could supply the needed information, whether a new scale needs to be created and collected from job experts or other relevant personnel, or whether information could be taken from other data sources, such as the Department of Labor and/or Occupational Outlook Handbook. Upon determining what types of data can be used for sources, it is necessary to determine the feasibility of obtaining this data. This is where the majority of unused sources are eliminated. For example, some sources may be appropriate for populating the key element but are not obtainable for use (i.e., limited access to information, copyrighted information, information that has yet to be collected in job analysis). Therefore, the final decision as to what sources will populate the metric is based on a combination of feasibility and necessity of including certain data. Steps six and seven in the process create an algorithm that combines the identified pieces of data in a way that will answer the target question. For the metrics described, statisticians developed various ways to combine metric data, specifically taking into account issues such as standardization, weighting and cut scores. These approaches were then evaluated, sometimes combined, and eventually configured into an initial formula. In many cases, the algorithms consist of procedural steps and equations with varying levels of complexity. After an algorithm is created, the eighth and last step in development prior to validation involves testing the algorithm using sample or normative data. Industrial/organizational psychologists should test the metric to determine whether the theoretical approach makes sense. During this phase of the process, flaws in theory may be discovered. For instance, it might be discovered that a normal distribution cannot be assumed for data on a particular metric. This is possible if the workforce analysis question targets a specialized group of workers whose technical expertise and ability is very similar or not likely to vary. The data may be specific enough to a particular job or job family so as not to require the assumption of normality. This is often the case with the use of job data that is unique to a particular job or organization. In addition, a review of the initial output by persons familiar with the job to determine whether the output makes logical sense. It may be determined through this evaluation process that the metric formula needs to be modified to better accommodate the data or more appropriately answer the workforce analysis question of interest. For example, suppose an algorithm was created to determine the important tools required for a job and was applied using data from a group of meteorologists. Examination of the output revealed that the most important tool was “weather balloon” while the least important tool was “weather radar system”. After reviewing the output, job experts indicate that this is an incorrect representation and in fact the tool identified as least important is much more important than the tool identified as most important. This examination of the algorithm and output revealed that the algorithm applied needs to be reevaluated and revised. After the algorithm has been tested and revised, another test using job data is needed. This data can be related to the set used to test the algorithm the first time or it can be from a different job. This process continues until the algorithm consistently produces logical and reliable output in that job experts are able to verify the output of the metric with their

316

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

expertise on the job. It may even be necessary to modify other steps in the metric development (e.g., key elements, sources, or combination of elements) to obtain results that target the workforce analysis question of interest (see Fig. 1). It is important to note that this process is not validation of the metric; it is only an initial check to determine if the metric is producing information aligned with its purpose. Full validation of any metric must be completed before it can be applied for decision making. 4.2. Metric development summary In the previous section, a comprehensive explanation of the metric development process was presented. The development of the metric Essential O⁎NET Skill Requirement for the Job is described in this section to further illustrate the metric development process. This description can be supplemented using the examples in Fig. 1. 4.2.1. Metric Example: Essential O⁎NET skill requirements for the job For every job there are necessary skills a worker must possess in order to perform effectively. Some of these skills are more critical, or more essential, for the job than other skills. Typical job analysis data provides practitioners with information about the skills used on the job, however, many times this set of skills is not bifurcated into those skills that are the most important for employee to possess versus the skills that are not critical to the performance of the job. As such, the main workforce analysis question that needs to be addressed (i.e., a metric needs to be developed) is “What are the essential skills needed to perform the job?” Identification of the essential skills for a job can be applied in areas such as selection and training/development. For example, understanding those skills that are essential to performing the job can aid practitioners in developing selection tools (e.g., structured interviews, simulations, skill proficiency tests) that ensure qualified individuals are selected. Using the output from this metric, selection tools can be developed to focus on those skills that an employee must have in order to be selected into the job. Those skills identified as essential in the output of this metric can provide a foundation upon which employers develop interviews. In addition, employers can use the information from the output of this metric to aid in determining which skills should be focused on in the interviewing stage of selection and which skills should be assessed through other means, such as tests. This can be accomplished by identification of those skills from the metric output that are best assessed through interviewing and those skills that are best assessed with other methods. This metric can also be used in conjunction with performance data on existing employees to determine the areas (skills) in which an individual needs training and/or development. Such performance data can be judged against those skills identified as essential to the job. The information revealed can help to identify those skills in which the individual needs development, thus allowing for tailored training programs to maximize the training effort. In order to determine the ‘essential skills needed for the job’, a list of key elements was first identified. The first element needed to answer this question is a list of skills. The next element needed is an importance level of the skill to the job. The final element needed is the frequency with which each skill is performed on the job. These three elements were selected because they were highly related to the workforce question of interest in that a combination of these elements have the potential to facilitate the desired applications to selection, training, promotion and other human resource areas described earlier. These elements were selected from a list that included other possible key elements such as skill performance difficulty, skill learning difficulty, and skill quality/utility. Having determined these key elements, a list of sources to populate them was established. Sources are options for how the user/developer will operationalize the key element. There are multiple sources for every key element. However, only a few will be feasible to obtain and appropriate for the metric under development. Source options considered for the first key element include the O⁎NET skill taxonomy, Bloom's Skill Taxonomy (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956), and occupational specific skills (i.e., skills that are not standardized across occupations but that are unique to the occupation or job being considered). Any of these options could provide some of the necessary information to populate the key element, a list of skills. However, the user should be primarily interested in the source that provides the most coverage in answering the workforce analysis question of interest. The O⁎NET skill taxonomy is potentially useful because it consists of 46 skills which can be divided into 2 lists: basic skills and cross-functional skills. Basic skills are skills that lead to the acquisition of other skills. Examples of basic skills include “writing” and “mathematics”. Cross-functional skills are performance relevant skills likely to be used on a variety of jobs. Examples of cross-functional skills include “instructing” and “troubleshooting”. Another important characteristic of using O⁎NET skills is that it is a standardized set of skills that can be used for comparisons across jobs.

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

317

The taxonomy of educational objectives, or Bloom's taxonomy, is another taxonomy considered as a source for this key element. This taxonomy was developed primarily for classifying, identifying, and creating test items for educational objectives (Krathwohl, 2002). In its revised form Bloom's taxonomy consists of 2 dimensions: cognitive and knowledge. Examples of cognitive skills include “application” “evaluation” and “creation”. Examples of knowledge skills include “factually-based skills” and “conceptually-based skills”. Though this taxonomy is not jobcentered or directly linked to occupational work, it still may be useful for identifying broader, non-technical skills. Another classification of skills considered was occupational specific skills. Occupational specific skills are a grouping of tasks that also include other job information like equipment and knowledges used on the job. This group of tasks is given a representative label that identifies the information within it. For example, an occupational specific skill for the job of meteorologist could be “Weather Radar Operations”. This specific skill would involve all tasks, equipment and knowledges relevant to weather radar operations. Occupational specific skills are potentially useful because they provide more job specific information about skills. However, they are often unstandardized and as such there is no way to utilize them in comparisons of skills across jobs. Given that Bloom's taxonomy is not specific enough to provide information to differentiate between jobs and that occupational specific skills are not standardized, it was decided that the O⁎NET skill taxonomy would be retained for use in this metric. The next key element needed to identify the essential skills for a job is an indication of the importance level of the skill to the job. There are various sources that could be used to reveal information about the importance of skills. Most of these sources include information about the tasks linked to skills and are obtained through job analysis data. Options include: information about task importance, consequence of task failure, dependency of task, demand of task, and difficulty of tasks. Further examination of each of these sources led to the conclusion that task importance operationalized by consequence of error during task performance was the most applicable piece of information for this metric. More specifically, it was determined that task dependency may not be a sound representation of task importance because a task can be important without being dependent on other tasks. Likewise, the level of task difficulty does not necessarily relate to the task's importance in that easily learned tasks may still be critical to job performance. In regards to demand for the skill, the importance of a skill cannot be determined by whether it is necessary upon entry into the job. In fact, many times the important tasks of a job are not heavily applied until the incumbent has been on the job for awhile. With these sources eliminated, the final two sources were ratings of task importance and task consequence of error. While a task importance rating could be used to tell us the importance of skills linked to tasks, it does not represent the positive or negative significance of not performing that task on the job. The consequence of error rating will not only reveal information about the task's importance but also about relationship between the tasks and effective job performance, (i.e., if these tasks are not performed properly, job performance severely suffers) thus telling us more about the skills important to the job. Therefore, task consequence of error rated by job incumbents was selected as the source for the second key element. The final key element needed is an indicator of the frequency with which the skill is used on the job. The source options for this key element include: a task frequency rating, a count of the number of tasks in the job by O⁎NET skill (e.g., Reading Comprehension is linked to 5 total job tasks, Written Expression is linked to 2 total job tasks) and a percentage of people in the job that use each O⁎NET skill. Two of the sources include information about the tasks linked to skills and are obtained through job analysis data. Information of particular importance in choosing the key element that will address the workforce analysis questions of interest here is the frequency of task application and the breadth of task use. Option one concerns the frequency component but not the breadth. Option three taps a version of breadth but is still not very informative. Option two does not address breadth in its current form, but can be expanded to incorporate both frequency and breadth. Frequency of application can be examined by identifying the count of the number of tasks in the job by O⁎NET skill. Breadth of use can be examined by modifying source option two into identifying the number of occupational specific skills linked to each O⁎NET skill. Occupational specific skills have not been included as a source up to this point in the metric. However, adding this component will add additional information in describing the frequency of skill use on the job. For further illustration of the key elements and sources identified for ‘essential skills needed for the job’, see Fig. 1. Now that the sources for populating the key elements have been identified, metric algorithm develop can begin. The purpose is to combine the three elements (i.e., O⁎NET skills, task consequence of error ratings, and task frequency ratings) in a way that is meaningful in answering the workforce analysis question. By using these sources for data the user is operationalizing “essential” as a combination of consequence of error and frequency of the tasks and how this

318

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

information relates to the O⁎NET skills used on the job. One algorithm considered for application involved two steps. First, all tasks in the job were rank ordered by consequence of error. This way the tasks that had the highest consequence of error could be identified. Second, the skill linked to each task was listed. The output revealed that the skills with the highest task consequence of error ratings and the most task linkages would be identified as most essential. It is important to note that a cut score would need to be created to differentiate the essential from non-essential skills. Though this algorithm would provide meaningful information about the skills on the job, it was not selected as the final algorithm for essential skills because it did not include the task frequency rating that provides meaningful information about how often a task, and therefore skill, is performed on the job. Another option for the essential skills algorithm is to first identify all the skills out of the standard O⁎NET taxonomy that are used on the job. This was conducted by extracting all skills for the job that had at least one O⁎NET skill linked to them. Then under each of these skills the task consequence of error was averaged. Next, the number of times each standard O⁎NET skill was linked to any of the occupational specific skills through tasks was identified. Therefore, the output of this metric was the product of the average task consequence of error under each O⁎NET skill ⁎ the number of tasks linked to the O⁎NET skill ⁎ the number of occupational specific skills linked to the O⁎NET skill. This algorithm was repeated for every O⁎NET skill used on the job. The final values for each O⁎NET skill were ranked ordered to form the output. This set of steps was recognized as the chosen algorithm for the essential skills metric because it provides information about both the importance of tasks and frequency/breadth of use. In order to assess whether the algorithm for essential skills produced interpretable output, the metric was applied using actual job data. In addition, the output was examined by individuals familiar with the job as well as individuals familiar with developing algorithms to analyze job data. The metric seemed to be differentiating essential from nonessential skills on the job as indicated by experts. Though this metric received initial support, it is important to note that this metric will be validated before application in real world settings. The output from this metric includes a rank ordered list of skills required for the job, and can be broken down into basic skills and cross functional skills. A cut off point should establish those skills that are essential for the job. This cutoff can be made at two different points to identify 3 different “bands” of skills, one band that is essential, one that is less essential, and one that is least essential to the job. This cutoff point will be established through validation. As previously mentioned, this metric output can be used for the development of selection tools, as well as for the development of training programs. 4.3. Metric validation Validation is an essential step in the development of any measure or tool that will be used to make personnel related decisions. Determining validity includes the establishment of a multitude of relationships to assess whether the item in question measures what it is designed to measure (McDonald, 1999; Messick, 1989; Whetzel & Wheaton, 1997). Establishing these relationships requires the accumulation of various types of information, both qualitative and quantitative, such as job performance scores and scores on other tests related to and not related to the new measure. When speaking of the validation of the metrics, the interest specifically lies in whether the output from the metric is an actual indicator of what the metric was intended to measure. Although there are a variety of approaches to validation, the approach for metric validation should concentrate specifically on construct validity and content validity. The systematic procedures used in the development of the metrics, such as the extensive use of existing literature related to key concepts used in the construction of each metric and the use of the O⁎NET framework as building blocks, contribute to initial content validity. However, further evidence bearing on the validity of these tools is needed prior to application in real world settings. 4.4. Implications Using metrics as a job data analysis tool presents several implications for researchers and HR practitioners alike. They can assist human resource professionals in addressing needs created by the lack of tools available to analyze job data discussed earlier in this article. Ultimately, with further development, testing, and validation, metrics have the potential to greatly impact the future of job data application and analysis. One meaningful implication is that when using metrics, data analysis output can be assessed in its entirety, rather than by sections. Data analysis output is often segmented because data are too massive and complex to assess as a

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

319

whole, especially with respect to qualitative techniques. Thus, when using job analysis output in workforce analysis and planning efforts, potentially important information may not be taken into account. This missing information could render any decisions using the job analysis data invalid, or at the very least biased. The use of O⁎NET and metrics ameliorate this occurrence by allowing the user to include all relevant data in the analyses. Another practical implication pertaining to the use of metrics is that they allow human resource professionals to keep themselves current as to the status of their workforce. When job information is quantified, tagged, and electronically stored, data are readily available for processing. As a result, job information is retrieved and processed at a faster rate. When using qualitative or even some quantitative methods, it can take months or even years for job analysis information to be processed and turned into meaningful information to be used in decision making (Urbanek, 1997). Part of this delay is due to the fact that the data are unique to the organization and the particular job or job family being analyzed. Thus, data may be years old before it is used, increasing the chances that when analysis takes place, the data are obsolete. Metrics make data analysis, and use of data, less cumbersome and more practical. Metrics also provide a method for conducting complicated analyses of job information at a lower cost to the customer. In industry, budget constraints often limit the type of analyses that can be conducted on job information. Job analysis procedures often require experts who are able to analyze and interpret data. The time and expertise of these individuals is costly. This cost is especially great when customized data interpretation is requested. Due to the high costs associated with expert interpretation, basic reports devoid of highly applicable conclusions are often requested by customers. Although these reports may provide supplementary information, they do not provide the kind of information needed to answer workforce questions relevant to the organization. Metrics automate complicated types of analysis and provide customization of interpretation without the use of analysts' time (and organizational/client money). Thus, the use of metrics has the potential to significantly reduce the costs involved in workforce analysis and planning. Additionally, the needs of human resource professionals are met in that they are able to use metrics to conduct more complicated analysis with their data and therefore aid in addressing more complicated organizational and job concerns. Metrics allow for the circulation of job analysis information to a variety of sources in an organization. As stated, human resource professionals often need their job analysis information to be portable across different parts of their organization. This is often not feasible when using other job analytic methods. Since job information can be easily misinterpreted by those that are less familiar with job analysis techniques, data interpretation is then left to those who have expertise in occupational analytics such as Industrial/Organizational psychologists. In addition, since interpretation requires certain specific knowledge, feedback from the data is often delivered only to the top tiers of an organization. Other areas of the organization that could benefit from feedback are often left out, or at the very least, only provided standard reports without the opportunity to customize reports to their specific needs. Metrics can aid in increasing the dissemination of job analytic information by allowing multiple individuals and departments within an organization to access and retrieve the information relevant to their concerns. For example, companies often allow each of their departments to define what experience requirements are necessary. These are typically referred to as skill sets or competencies. The problem with this practice is that the information defined by each department can vary so drastically that it is difficult to consider employees across departments for internal transfers or hires. Consider the difference between an accounting and marketing department. An accounting department may require their employees to have basic mathematics skills, accounting and computer knowledge, whereas a marketing department expresses the need for their employees to have customer service, organization, and leadership skills. This lack of standardization poses potential problems for the organization by making it difficult to know whether an employee with relevant background can transfer from accounting to marketing or vice versa. This departmental distinction also makes promoting an employee to management complicated because it requires comparing two employees from two different departments. It also makes it difficult to find shared skills that can be trained or emphasized consistently company wide. Using metrics with standardized data from sources like O⁎NET allows for the comparison of individuals within the organization on the same elements. Therefore, decisions regarding transfer can be made after reviewing the qualifications of each individual using the same point of reference. This increases fairness in selection and permits the transfer or promotion of the most appropriate employee. Furthermore, standardized information can help organizations analyze information across their industry. For example, if every job was defined by the same set of skills, abilities, and work duties, a company might find comparable occupations from which to recruit that were previously untapped. For instance, if an examination of an

320

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

industry reveals that foodservice employees have the same general skillset as employees of customer service centers, this may introduce a new avenue for recruiting in an organization. Finally, as mentioned in the introduction, organizations exist in environments that are often unpredictable. The utilization of O⁎NET in conjunction with metrics allows organizations to use job analytic information to respond to the demands of increasingly dynamic organizational environments. Many times, job analysis information is poorly collected, analyzed and interpreted. Often obsolete data is utilized in order to make current or strategic organizational decisions. Not lost in this point is the fact that an organization's only alternative to using this deficient job data would be to re-conduct job analyses for the workforce. Despite the importance of this job data to the functioning of organizations, organizations are often unwilling to apply the financial resources required to complete such an effort. Using O⁎NET, organizations can continually update job analytic information. This is plausible because O⁎NET uses existing taxonomies (i.e., knowledges, skills, abilities) on which information is gathered and stored electronically. The use of metrics and O⁎NET provides organizations with an advantage in that they are able to answer person, job or workforce specific questions. Additionally, organizational decision makers can be more accurate when they are trying to match organizational goals structure and production with that of their changing environment. 4.5. Future of metrics Though the metrics that are currently developed address traditional workforce areas such as recruitment, selection, promotion, and training, there is considerable room for expansion of metrics into other relevant workforce areas. Other organizational areas that have the potential to benefit from the use of metrics include teamwork and leadership. Additionally, the future of metrics involves implementing online, automated systems for use by HR professionals. As mentioned, organizations are moving toward the use of cross-functional, highly independent teams. Incumbents often are assembled into work teams to facilitate performance and increase knowledge of other's jobs. As a result, team factors are emerging as an important area of performance for assessment. Researchers have identified five individual level characteristics relevant to team performance. These include team orientation, mutual performance monitoring, back-up behaviors, team leadership, and adaptability/flexibility (Cannon-Bowers & Salas, 1997). These and other team characteristics could be assessed at an employee level and analyzed to make decisions about work team formation in an organization. Additionally, assessing performance of these characteristics using metrics and providing feedback to employees could facilitate employee development and therefore increase team performance. Another area of future metric development is leadership. Leadership metrics could be used to determine individual employee leadership performance or the capability of individuals to perform well in leadership roles. Researchers argue that a major contributor to effective leadership is an individual's skill in solving problems (Mumford, Zaccaro, Johnson, Diana, & Threadfall, 2000). This skill based approach to leadership gives rise to the possibility for metrics that use an individual's proficiency in problem solving skills as an indicator of their leadership potential. Metrics such as these could be used in conjunction with promotion or transferability metrics to ascertain which employees are best suited for upper level managerial positions in the organization. Metrics can be created or adapted to address most any area of HRM. Teamwork and leadership are just two of the untapped areas that could benefit from the use of metrics. If relevant organization, job, or individual level data are collected or exist currently, metrics can be used to effectively analyze it for decision making. Currently, only the logic, formulas, and output interpretation of the metrics described here are available to users. As stated, any metric should undergo a full content and construct validation prior to being used with job data to make human resource decisions. All metrics described in this article and listed in Appendix A will be validated as a part of an ongoing effort with SkillsNET® Corporation and the Department of Defense. In the future, metrics could be in an online, automated format promoting ease of use and time management. Users would then be able to simply select the job data and metric(s) they would like to use, click a button, and print the output. This online format will only further advance the capability of metrics for use in organizations. 5. Conclusion Job analysis information is a critical component in many HRM functions. The data obtained from job analysis can aid in decision making for the areas of recruitment, selection, promotion, transfer, training, individual and

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

321

organizational development, and performance appraisal. These decisions made by human resource professionals are becoming more critical to the successful functioning of the organization because of various changes in the workforce. Worker requirements are transforming due to prevalence of cross-functional work teams. Distinctions between supervisors and subordinates are disappearing as organizational structures become flatter and communication occurs across many levels. These changes have led to the need for different types and larger amounts of job analysis data than traditionally obtained. The collection of job analysis data is now performed with relative ease due to the increased use of technology in data collection efforts. With the use of technology, large amounts of data can be collected from multiple individuals in shorter amounts of time. Thus, databases to house the data are typically very large and complex. Due to the sheer size and various types of data stored in these databases, it is very difficult to effectively utilize the data. Many times data are not used to full potential because of a lack of appropriate methods to do so. Furthermore, without standardized elements in the job data used for analysis, decisions made using the data are not applicable across jobs or departments within an organization. Metrics offer an advanced yet user-friendly methodology that allows job data to be used to answer questions related to HRM. In conjunction with O⁎NET and unique job data, metrics are innovative tools designed to aid HRM professionals in making more objective decisions. Metrics introduce capabilities for job data that have previously been unattainable when using qualitative and less advanced quantitative methods. Ultimately, metrics allow organizations and HR professionals to embrace the changes resulting from the evolving world of work with an equally advanced analysis tool.

Appendix A. Developed metrics

Name of metric Job analysis metrics Essential knowledge requirements for the job Essential O⁎NET ability requirements for the job Essential O⁎NET skill requirements for the job Essential tool requirements for the job Estimated task importance Job essential occupational specific skills Task variety O⁎NET skill variety Cognitive demands of job Degree of technological skills needed on the job Complex job descriptors Job complexity Physical demand Psychosensory demands of a job Job difficulty Stress demands on job Temporal change Temporal job proficiency change Job essential occupational specific skill complexity Job delay tolerance Level of expertise required

Objective of metric To determine the knowledges that are essential to performing the job To determine the abilities that are essential to performing the job To determine the O⁎NET that are essential to performing the job To determine the tools that are essential to the job To determine the importance of a task in the absence of a specific rating of task importance To determine the occupational specific skills and tasks essential to accomplishing a job To determine the degree to which tasks performed on the job are performed in a repetitive manner To determine the degree to which a variety of skills are used on the job To determine the degree to which the job is cognitively demanding To determine the degree to which a job requires technological skills for adequate performance

To determine the complexity involved in performing the tasks of the job To determine the degree to which the job is physically demanding To determine the degree to which the target job requires psychomotor and sensory abilities To determine the degree of difficulty related to performing the job To determine the degree to which the job is stressfully demanding To identify significant changes in job characteristics over time To identify changes in the proficiency levels needed to adequately perform a job To determine the complexity involved in performing the tasks associated with the Job Essential Occupational Specific Skills To determine the degree to which an organization can function in the absence of a particular job To determine the overall level of expertise necessary to properly perform the skills associated with a job (continued on next page)

322

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

Appendix A (continued ) Level of proficiency required

Recruitment, selection, and transferability metrics General difficulty of external recruiting for a job General difficulty of internal recruiting for a job Immediate proficiency needs Key capabilities needed for the job Eligibility for promotion Personnel overage requirements

Job family metrics Transferability of people across positions Transferability of people across jobs Transferability of jobs Difficulty of replacement Skill, ability, and GWA similarity Skill set required for emerging job Similarity of jobs based on content (non-O⁎NET jobs)

Training metrics Cost of training Entry level of transfer employee training time Ease of training implementation Material to be trained On-the-job trainability of job essential occupational specific skills Overall on-the-job trainability Estimated task learning difficulty Degree of job technical learning difficulty Job learning difficulty based on rating Estimated job learning difficulty

Mission/military specific metrics Criticality of job to the mission Degree of required teamwork Essential skills for the mission Person fit for the mission Likelihood of mission detrimental error Number of people required for the mission

To determine the overall level of proficiency to properly perform the skills associated with a job

To determine the level of difficulty associated with recruiting individuals from outside the organization for a job To determine the level of difficulty associated with recruiting individuals from inside the organization for a job To determine the degree to which a job requires complete skill and task proficiency upon entry To determine the key capabilities that the employee must have in order to be able to perform the target job To determine if a person fulfills the requirements to move to a higher position To determine the degree to which additional personnel are required to be available to cover the functions of a job under adverse conditions and high demand

To determine the degree to which a person in a current position is able to fulfill the requirements of a target position To determine the degree to which a person in a current job is able to fulfill the requirements of a target job To determine the degree to which job components/requirements can be fulfilled by the components of another job To determine the degree of difficulty involved with finding a person to fill a job To determine the degree of similarity between two jobs that have data available in the O⁎NET database To determine the essential skills needed for a new job with no prior connected job data To determine the degree of similarity between two jobs that do not have data available in the O⁎NET database

To determine the cost involved in training the job To determine the amount of time it takes to train an entry level or transfer employee to a required level of proficiency in a job To determine the relative difficulty of training program for a job To determine what material within a job is essential to train in order for the job to be successfully performed To determine the degree to which the job essential occupational specific skills can be trained using on-the-job training methods To determine the degree to which the job essential tasks can be trained using on-the-job training methods To determine the learning difficulty of a task in the absence of a specific rating of task learning difficulty To determine the degree of difficulty required to learn the how to use the tools associated with a job To determine the amount of time it takes to learn the occupational specific skills associated with a given job To determine the amount of time it takes to learn the occupational specific skills associated with a given job

To determine the criticality of the job to the mission To determine the degree to which the tasks performed on the job require coordination with other people To determine the essential skills across all jobs in the mission To determine the requirements needed for a mission and then determine the best capable person(s) to contribute successfully to a mission To determine the likelihood of making a serious mistake on the job that may negatively impact the mission To determine the number of people needed to successfully complete the mission

L.E. McEntire et al. / Human Resource Management Review 16 (2006) 310–323

323

References Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. New York: David McKay. Borman, W. C. (1996). The occupational information network: An updated dictionary of occupational titles. Military Psychology, 83, 263−265. Buckley, P., Minette, K., Joy, D., & Michaels, J. (2004). The use of an automated employment recruiting and screening system for temporary professional employees: A case study. Human Resource Management, 43, 233−241. Cannon-Bowers, J. A., & Salas, E. (1997). Teamwork competencies: The interaction of team member knowledge, skills, and attitudes. In H. F. O'Niel (Ed.), Workforce readiness: Competencies and assessment (pp. 151−174). Mahweh, NJ: Lawrence Erlbaum Associates. Cascio, W. F. (1995). Whither industrial and organizational psychology in a changing world of work? American Psychologist, 50, 928−939. Chermack, T. J. (2003). Decision-making expertise at the core of human resource development. Advances in Developing Human Resources, 5, 365−377. Crespin, T. R., & Austin, J. T. (2002). Computer technology applications in industrial and organizational psychology. CyberPsychology and Behavior, 5, 279−303. Cunningham, J. W. (1996). Generic job descriptors: A likely direction in occupational analysis. Military Psychology, 8, 247−262. Dessler, G. (2003). Human resource management. Upper Saddle River, New Jersey: Pearson Education. Harvey, R. J. (1991). Job analysis. In M. D. Dunnette b L.M. Hough (Eds.), Handbook of industrial and organizational psychology, vol. 2 (2nd ed.) (pp. 71−163). Palo Alto, CA: Consulting Psychologists Press Inc. Harvey, R. J., & Wilson, M. A. (2000). Yes Virginia, there is an objective reality in job analysis. Journal of Organizational Behavior, 21, 829−855. Jeanneret, P. R., & Strong, M. H. (2003). Linking O⁎NET job analysis information to job requirement predictors: An O⁎NET application. Personnel Psychology, 56, 423−429. Korte, R. F. (2003). Biases in decision making and implications for human resource development. Advances in Developing Human Resources, 5, 440−457. Krathwohl, D. (2002). A revision of Bloom's taxonomy: An overview. Theory into Practice, 41, 212−218. Mariani, M. (1999, Spring). Replace with a database: O⁎NET replaces the dictionary of occupational titles. Occupational Outlook Quarterly, 3−9. Mariani, M. (2001, Fall). O⁎NET update. Occupational Outlook Quarterly, 26−27. McCann, J. (2004). Organizational effectiveness: Changing concepts for changing environments. Human Resource Planning, 27, 42−50. McCormick, E. J. (1976). Job and task analysis. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp. 651−696). Chicago: Rand McNally. McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum Associates. McDonald, E., & Cornetto, M. (2005). Putting HR data to work is easier than ever. Employee Benefit News, 19, 18−19. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13−103). New York, NY: Macmillan Publishing Co, Inc. Mumford, M. D., & Peterson, N. G. (1999). The O⁎NET content model: Structural considerations in describing jobs. In N. G. Peterson, M. D. Mumford, W. C. Borman, P. R. Jeanneret, & E. A. Fleishman (Eds.), An occupational information system for the 21st century: The development of O⁎NET (pp. 21−30). Washington, DC: American Psychological Association. Mumford, M. D., Zaccaro, S. J., Johnson, J. F., Diana, M., & Threadfall, K. V. (2000). Patterns of leader characteristics: Implications for performance and development. Leadership Quarterly, 11, 115−133. Murphy, T. E., & Zandvakili, S. (2000). Data and metrics driven approach to human resource practices: Using customers, employees and financial metrics. Human Resource Management, 39, 93−105. Patterson, B., & Lindsey, S. (2003, September). Mining the gold: Gain competitive advantage through HR data analysis. HR Magazine, 48, 131−136. Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., & Fleishman, E. A. (1999). An occupational information system for the 21st century: The development of O⁎NET. Washington, DC: American Psychological Association. Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., Fleishman, E. A., Levin, K. Y., et al. (2001). Understanding work using the Occupational Information Network (O⁎NET). Personnel Psychology, 54, 451−492. Riggio, R. E. (1990). Introduction to industrial/organizational psychology. Glenview, IL: Scott, Foresman and Co. Sanchez, J. I. (1994). From documentation to innovation: Reshaping job analysis to meet emerging business needs. Human Resource Management Review, 4, 51−74. Sanchez, J. I., & Levine, E. L. (2000). Accuracy or consequential validity: Which is the better standard for job analysis data? Journal of Organizational Behavior, 21, 209−220. Schneider, B., & Konz, A. M. (1989). Strategic job analysis. Human Resource Management Review, 28, 51−63. Siddique, C. M. (2004). Job analysis: A strategic human resource management practice. International Journal of Human Resource Management, 15, 219−244. Urbanek, S. J. (1997). Job analysis: A local government's experience. Public Personnel Management, 26, 423−430. U.S. Department of Labor (1991). Dictionary of occupational titles (4th ed.). Washington, DC: Author. Walker, J. W. (1990). Human resource planning, 1990s style. Human Resource Planning, 13, 229−240. Walker, J. W. (1999). Is HR ready for the 21st century? Human Resource Planning, 22, 5−7. Whetzel, D. L., & Wheaton, G. R. (1997). Applied measurement methods in industrial psychology. Palo Alto, CA: Davies-Black Publishing.