Picture this! Multivariate analysis in organizational assessment

Evaluation

and Program

Pergamon

Planning. Vol. 17,

No. I, pp. 19-24, 1994 Copyright 0 1994 Elsevier Science Ltd Primed in the USA.All right\ reserved

0149.7189/94 $6.00+ .oo

PICTURE THIS! MULTIVARIATE ANALYSIS IN ORGANIZATIONAL ASSESSMENT DANIEL J. MCLINDEN and DARRYL L. JINKERSON Center

for Professional

Education,

The Arthur

Andersen

Worldwide

Organization

ABSTRACT Finding the proper level of analysis for quantitative data is a problem in organizational assessment. First, the use of simple descriptive statistics often fails to adequate/y portray relutionships between variables. Second, although inferent~ai statistics can s~~rnt~~ari~e data and better describe the relationships between variables, these techniques are often based on a-priori decisions regarding specific group comparisons within the data. Such a-priori decisions may be inconsistent with respondent views. On the other hand, multivariate methods which overcome the previously cited limitations and allow an exploration of the structure of the data tend to be quite cotnplex and difficult for clients to understand and interpret. Some multivariate methods can portray results in a more graphical,form or perceptual map. This approach to analysis allows both the clients and stakeholders to ‘picture” the results of complex statistical analysis. One such technique and the.focus of the current paper is Multidirne~~s~ona~Unfolding (MDU). The intent is to highlight the utifit_vof mapping data and advo~atefurther a~~ficat~ons of mapping studie.9 in analyzing and l~ndersianding organi~af~ona~ issues. Consultants charged with assessing organizations often find themselves in the position of having to communicate quantitative findings to clients who often do not possess specific analytical expertise. That is, the audience for the results is often made up of executives, program administrators and other stakeholders whose area of expertise lies outside the realm of research and statistics. A communication problem can arise in such situations. For instance, the consultant may believe that the data contain complex relationships between variables which, if properly identified, could lead to a more informed decision by the client. However, the client lacking a sophisticated quantitative background may be biased to a more basic analytical approach such as through averages, frequencies, and correlations. A client-centered approach in this situation encounters two problems. First, if many variables are involved, it may be unreasonable to expect a client to properly grasp the relationship behind a large array of numbers. For example, consider results which are comprised of the average ratings on a SO-item questionnaire which is also broken down by five personnel levels. The resulting table would contain 250 points (five personnel lev-

els by 50 items) of information. Certainly the shades of meaning would be lost in attempting to draw conclusions from such a table. Second, selecting specific variables individually or several at a time may fail to portray the overarching relationships between groups of variables. A solution to these challenges is the use of multivariate techniques. However, multivariate statistics, while capable of highlighting intricate relationships, are generally quite complex and in our experience tend to make clients uneasy. This uneasiness arises from the fact that the computer printouts contain unfamiliar terms and are generally not intelligible except by an expert. In this scenario the client is, at best, left to base decisions solely on trust in the consultant as expert or, at worst, to ignore the findings altogether. An alternative and more acceptable approach is to make the results readily accessible to the client while also retaining the complexity of information. Accessibility can be achieved by summarizing the underlying structure of the data and providing this summary in a graphical format. Several multivariate statistical approaches are well-suited to presenting analyses in this fashion. Perhaps the best known is Multidimen-

Requests for reprints should be sent to Daniel J. McLinden, tion, 1405 North Fifth Avenue, St. Charles, IL 60174.

Andersen

The Arthur

19

Worldwide

Organization,

Center

for Professional

Educa-

20

DANIEL

J. McLINDEN

sional Scaling (MDS) which was aptly described by Young (1987): Multidimensional scaling (MDS) rests on the premise that a picture is worth a thousand numbers. The term multidimensionalscaling refers to a family of data analysis methods, all of which portray the data’s structure in a spatial fashion easily assimilated by the relatively untrained human eye. That is, they construct a geometric representation of the data, usually in a Euclidean space of fairly low dimensionality. (p. 3) Owing to the ability to represent the complexity of multivariate data in a conceptually straightforward fashion, MDS and related techniques have received substantial attention in applied work. While scaling methods have emerged historically from psychological disciplines, our belief is that much of the application in solving business problems has been in marketing research (e.g., see Green, Carmone, & Smith, 1989). A notable exception has been the work on concept mapping (Trochim, 1989a, 1989b). Additional opportunities exist to expand the utility of mapping through the use of related techniques. In particular, Multidimensional Unfolding (MDU) provides the capability to explore relationships among variables in a “joint space.” That is, the map can simultaneously portray relationships between specific variables, such as questionnaire items and demographic information regarding the respondents who completed the instrument. We have found this approach to data analysis to be useful in several cases and believe that additional opportunities exist in the practice of organizational consulting to expand the utility of these techniques. As such, the focus of this paper is to describe two case studies that illustrate the application of these methods to specific organizational concerns. MULTIDIMENSIONAL

UNFOLDING

Unfolding analysis begins with a rectangular data matrix. For example, the rows could be cases (e.g., survey respondents) and the columns could be variables (see Figure 1, part A). The MDU algorithm then attempts to fit each variable and each case into a joint spatial arrangement. For instance, in a two-dimensional solution each case and each variable would have specific coordinates indicating their location along both the x- and y-axes. One set of points would represent the rows in the data matrix, which in this instance would be cases. A second set of points would represent the columns in the data matrix which would be the variables (see Figure 1, part B). Variable points which are close in proximity would be viewed as being perceived by the respondents as similar. Analysis would be based on an examination of the geography of the map or, in other words, the spatial arrangement of points (see Figure 1, part C). For example, groups of points (i.e., points arranged in close proximity to each other) could be interpreted as express-

and DARRYL

L. JINKERSON

ing a common theme. Additionally, points for cases would then be examined to determine with which variable(s) they are most closely aligned. Close proximity of a case point to a variable point would indicate that the individual had a strong preference for the construct expressed by the variable. In sum, proximity between points indicates the strength of a relationship. Where points coalesce into groups, common themes among the traits can be identified and the group can be labeled with a single overarching theme. Finally, in addition to examining the geography of the map, a dimensional interpretation can be considered. A determination should be made whether issues vary systematically along the vertical and horizontal dimensions. However, while interpreting the dimensions may help to explain relationships in the data, some caution is necessary. Specifically, the x- and y-axes may be drawn in any orthogonal position through the origin on the map. As such, the vertical and horizontal dimensions which result from a computer printout are not the only possible orientations for the axes. APPLICATIONS Retention of Health-Care Personnel A secondary analysis was conducted on a survey designed to assist a government agency in assessing problems relative to the retention of health-care personnel. An SO-item survey questionnaire was completed by a random sample of 2,537 individuals in the health-care industry. Thirty-six of the questions asked respondents to rate their degree of agreement with statements about their attitude toward work and conditions of employment. Items dealt with such issues as amount of work, recognition, rewards, overtime, and so on. Our goal was to examine the relationships between items and personnel classes. As such, an intermediate step prior to analysis was to summarize the raw data. Means were computed for each item within each personnel class. The result was a matrix containing one row for each personnel class, or a total of six rows, and one column for each variable, which in this case numbered 36. Each cell in the 6 x 36 matrix then contained the mean rating for a personnel class on a specific questionnaire item. Multidimensional Unfolding produced a two-dimensional map. Stress, which is an indicator of the goodness of fit between the data and the resulting map was at an acceptably low value (0.10) with two dimensions. Given that three dimensions are more difficult to visualize and interpret and that the stress was acceptable with the two dimensional solution, the two dimensional solution was used for interpretation. Since the resulting map contained a large number of points, hierarchical cluster analysis (average linkage) was applied to the coordinates to group the points into a smaller set of clusters

Picture

This!

21

Part 8:

Part A:

Par) C:

Figure 1. Data structure: From raw data to map.

in order to make the map easier to interpret (see Figure 2). A six cluster solution was chosen to represent the data (R’ = 0.86). While more clusters would have increased the value of R2,the six cluster solution seemed

to produce interpretable clusters and kept the number of clusters to a minimum in an effort to balance the need for simplicity (i.e., fewer clusters) with the need to account for a reasonably high proportion of the variance.

f RETENTIONSTUDY 3

3

)

I

UNDERSTANDlNC THE ORGANIZATION

SUF’F’ORT PERSONAL DECISION MAKING

- 2

t

Figure 2. Multidimensional

space for the survey of health-care

practitioners.

22

DANIEL

J. M~L.~NDEN

The resulting map (Figure 2) shows each of the points which represent the 36 questions and the six points which represent each of the personnel levels. These results indicate that respondents considered six issues in evaluating their working conditions. In a practical sense, focusing on six issues rather than 36 questions makes the task of moving from information to decision making easier because neither the client nor consultant is overwhelmed by data. Group points are oriented toward those issues which the respondents rated most favorably. The six personnel classifications are all oriented toward the positive side of the x-axis and within close proximity to a large cluster of items labeled “Professional Support.” This cluster encompassed items which implied professional respect, mutual support, and communication. Items in this cluster were those which respondents viewed most positively. At the other extreme of the x-axis were items which respondents viewed negatively (pay, overtime, assigned duties, and so on). In sum, respondents generally felt they were underpaid and overworked. Differences between the groups were apparent when examining the vertical dimension. Ah of the trainees were oriented toward the negative side of the y-axis, while professional staff were located in the positive area. The trainee groups were oriented toward items indicating satisfaction with their jobs and the support they received from superiors. Professional staff, on the other hand, were more closely aligned with items pertaining to understanding the organization. The exception to this was the group at approximateIy the zero point on the y-axis. This group was the most highly educated and skihed. These more seasoned employees seemed, like the trainees, to place a higher value on the need for a supportive environment, The key feature of this analysis is that viewing the data as portrayed in Figure 2 enables the decision-maker to identify overarching issues as opposed to simply focusing on individual items. Further, the map enhances the ability to determine which issues are most salient for each group. Finally and perhaps most importantly, a client or stakeholder ought to be capable of participating in this interpretation. Communication Patterns Multidimensional Unfolding was employed as one of a series of analytical approaches to examine communication effectiveness within an international organization. A representative sample of participants from six personnel classification levels from throughout the world (n = 1,320) was surveyed regarding overall effectiveness of internal communication, communication needs, current communication channels/sources, suggestions for improvements, and strategies for measuring successful communication. The section of the instrument analyzed using MDU dealt with the quality of information received from oth-

and DARRYL

L. JINK~RS~~

ers. This section consisted of 22 topics for which an individual receives information on subjects such as personal performance, project responsibilities, employee benefits, promotion, salary, and so on. Respondents completed the following two questions for each of these topics: (a) How much information do you actually receive? and (b) How much information would you like to receive? Each question was rated on a six-point Likert scale (0 = not applicable, 1 = very littIe, 5 = very much). From the standpoint of organizational effectiveness, an optimal response occurred when the difference between the two questions was zero. Negative or positive difference values would, respectively, indicate an information underload or overload. In determining with our client the statistical approaches to be used in each section of the communication survey, it became apparent that the client felt strongly that an MDU type of approach was necessary. Our client in this study, having been exposed to perceptual mapping via prior marketing experience, was very keen to demonstrate to his constituency the merits of this approach. Of particular interest to him was the technique’s ability to “paint a picture” of the results. His wish, coupled with our desire to replicate the approach developed in the prior project, resulted in our second application of mapping. Of particular interest to our client was graphically portraying the relationship between personnel classification (n = 4) and the 22 communication topics. The mean difference rating (amount received less amount needed) was calculated for each topic within each of the personne1 levels. That is, a matrix of the means was developed as described in the prior study. In this study, both an under-load and an overload were seen as equally undesirable for the organization. As a result of this viewpoint, tJ~e absolute value of the difference scores was calculated and used as the input dataset. The analysis resulted in a two-dimensional solution with a stress value of 0.07 (see Figure 3). In interpreting Figure 3, the first step was to identify those topics which were close together and then to identify the theme represented by those topics. Unlike the healthcare study, fewer points were plotted; thus we felt confident that the graph could be interpreted directly and chose not to utilize cfuster anaIysis. The groups were labeled Performa~l~e, Business Information, Competitors, Failures, Programs, and Career Path. The previous study was confined to a geographic analysis of the map; however in this study a dimensional interpretation seemed appropriate. While we were cognizant of the risks involved in a dimensional interpretation, in examining the map we concluded that the apparent axes represented a reasonable basis for further interpretation. Therefore, the next step in the interpretation was to label or name the vertical and horizontal dimensions. Specifically, the task was to determine if the spatial ar-

4

I

INTERNAL

-3

I I

Figure 3. Multidimensional

-2

,

LEVEL2 .

t

.

LEVEL

BUSI~E$S INFORMATION PROP SOWWARE PROJECT

issues.

TECH DEVELOPMENTS BUSINESS DIREtXION SERVICE LINE PROGRAMS

1

space for the survey of communication

IMPACT

-4 FUTURE

-3

L

2

CURRENT

24

DANIEL J. McLINDEN

and DARRYL L. JINKERSON

rangement of items constituted an order which could be described on each dimension. After considering the location and nature of each cluster, the x-axis was labeled “organization locus of control” and the y-axis was labeled “impact.” We determined that in moving from left to right on the x-axis, the issues involved first internal issues and then external issues. Likewise, in moving from top to bottom on the y-axis, the issues were first concerned with current events and then future events. The position of the four personnel classifications was examined relative to the clusters. Lower personnel classes (e.g., entry level positions) received sufficient communication on performance but were the farthest from receiving the optimal amount on most other issues. Likewise, Level 2, which is only slightly higher on the career ladder, was positioned relatively far from optimal information on most issues. Higher personnel classes, middle management and executives (Levels 3 and 4) were closer to the optimal amount of information, especially regarding business information and programs. Finally, the results of the analysis suggested that no group was receiving the optimal amount of information regarding either company competitors or company failures. DISCUSSION While the results of this type of analysis seem useful, the question remains: Can the results be considered a basis for decision making? While the answer to the question is “yes,” some caution is advisable since MDU tends to present several problems (Davison, 1983). First, the process may stop at a solution, termed local minima, which satisfies the equations involved but does not obtain the optimal fit. Second, the solution may degenerate; that is, many variables will be presented in space by only a few overlapping points. Third, the analysis may fail to converge to a solution. Of the three, the problems of local minima and degenerate solutions have been cited as particularly problematic in unfolding analysis (Schiffman, Reynolds, & Young, 1981). We have considered the limitations associated with MDU and while caution is prudent in pursuing this type of analysis, this approach to organizational assessment has much to offer. However, additional research is needed. In particular, research involving new applications is warranted. While we have highlighted several cases here, much remains to be learned about achieving the ultimate advantage of these methods in serving the interests of our clients. Of particular interest would be additional case studies. These techniques can certainly be expanded to other applications of organizational assessment. Additionally, the application of other methods to similar problems is also warranted. Future applications might include the use of other techniques such as principal components analysis, discriminant analysis and so on. In some cases the techniques are very similar,

since they are derived from a similar mathematical framework and likely to produce a similar solution. In other cases different techniques are better suited to some tasks, not to others. For instance, we have applied correspondence analysis to some problems of portraying client perceptions in a joint space when categorical data is the basis for decision making. In the meantime, however, these techniques do offer the organizational consultant a useful tool and, if prudently applied, the results can increase the utility of a research project. Mapping techniques such as MDU, as well as other techniques, hold promise for situations in which quantitative data must be collected and a consultant needs to portray complex information to a diverse audience. Mapping can provide a means of drawing together in a single picture the multivariate nature of a data collection instrument. For example, in the cases cited here the data from responses to questions and the types of respondents were all portrayed in the same graphic representation. Thus, these techniques offer a chance to provide a comprehensive analysis of multiple variables. In addition to being an analytical tool, this approach offers an opportunity to increase the value of organizational studies in two ways. First, given the graphic nature of the results, interpretation can be shared among consultant, client, and other stakeholders. This allows the evaluation consuItant to utilize analyses which address the complexity of the data. At the same time, other stakeholders, particularly those who may lack expertise in statistics can review, understand, and challenge results. Second, this tool provides a way to identify emerging issues. Rather than quantifying pre-defined issues, the interpretive process enables issues to arise from the data. In the examples in this paper, the clusters and their relationships to personnel levels were not anticipated results. As such, this approach seems well suited to situations where the evaluator and stakeholders are exploring data and attempting to define issues. REFERENCES DAVISON, M.L. (I 983). hluliidi~nertsioPruiscil/ing. New York: John Wiley & Sons. GREEN,

P.E.,

CARMONE,

F.J., 8: SMITH,

S.S4. (1989). iM~d/riAllyn &

dimensional scaling: Concepls and upplications. Boston: Bacon. SCHIFFMAN,

S.S., REYNOLDS,

M.L.,

& YOUNG,

F.W. (1981).

Iniroductron IQ muiiidimensionul scaling: Tileory, methods, and applicaiiota. New York: Academic Press. TROCHl&4. for planning l-16.

W.M.K. (1989a). An introdliction to concept mapping in evaluation. Evaiuation and Program Plunnirfg, I2,

TROCHIM, W.M.K. (1989b). Concept mapping: Soft science or hard art? Evaluation und Program Plurfning, 12, 87-I IO. YOUNG,

F. W. (1987). Mulridimet7sional scakng: History, fl7eory and NJ: Lawrence Erlbaum Associates, Inc.

upp/icu/ions. Hillsdale,