A study on the consistency of stakeholder preferences for different types of information in evaluating police services

Evaluation and Program Planning, Vol. 9, pp. 13-24,1986

0149-7189186 $3.00 + .OO Copyright o 1986 Pergamon Journals Ltd.

Printed in the USA. All rights reserved.

A STUDY ON THE CONSISTENCY OF STAKEHOLDER PREFERENCES FOR DIFFERENT TYPES OF INFORMATION IN EVALUATING POLICE SERVICES STUARTJAY DEUTSCH Georgia Institute of Technology

CHARLESJ. MALMBORG Rensselaer Polytechnic Institute and State University

ABSTRACT A questionnaire k designed to allow assessment of a simple additive valuefunction for testing respondent preferences for different types of information used in evaluating police services. The questionnaire was dhtributed to 57 respondents representingfive different stakeholder groups within the metropolitan Pittsburgh Criminal Justice System. The responsesare analyzed to determine what types of information different stakeholder groups consider useful in evaluating police performance. Statistical analysis of results suggest that different stakeholders have a high degree of constitency in the importance attached to different types of information. within the CJS maintain different preferences with respect to evaluating police services. This is accomplished using a multi-attribute approach for assessment of the relative importance of measures relating to different objectives of police services from a predetermined set of measures. The matrix model of Deutsch and Malmborg (1982a, 1982b, 1982c) is adapted to develop a value function yielding a prioritization of measures relating to different objectives and this functional form is assessed for different stakeholder groups within the CJS. Use of a multi-attribute approach requires decision makers (i.e., stakeholders) to systematically consider elements of the evaluation problem that have been specified by stakeholders prior to the assessment of the importance of objectives. This creates a common framework by which the priorities expressed by different stakeholders can be compared. In the next section, the advantages of using a multiattribute approach are discussed and the appropriate variation of the matrix model is introduced. The third section describes the pre-established objective set and the typical performance measures associated with each

The evaluation of police services is complicated by the presence of multiple evaluating constituencies within the Criminal Justice System (CJS). Each of these different evaluating constituencies, referred to as stakeholders, maintains a different degree of penetration into the CJS and a different sphere of influence with respect to evaluating police services. Connolly, Conlon, and Deutsch (1980) argue that determination of a single measure of merit for organizational effectiveness is not apt to yield a valid measure of evaluation. At least in part, this may be due to the fact that different stakeholders within the CJS, or any other organization, maintain different objectives and thus consider different measures of organizational effectiveness to be relevant. Thus, the evaluation of police services may reflect the objectives of a particular stakeholder group if such groups do not maintain the same set of objectives. This possibility has serious implications for the design of an information system to be used in the evaluation of police services. The purpose of this study is to perform an empirical test of the hypothesis that different stakeholder groups

This work was performed under Grant No. 78-NI-AX-0040 from the National Institute of Justice. Points of view or opinions stated herein are those of the authors and do not necessarily represent the official position or policies of the United States Department of Justice. Requests for reprints should be addressed to Stuart .I. Deutsch, School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332. 13

14

STUART JAY DEUTSCH

and CHARLES J. MALMBORG

objective. The development of an assessment questionnaire is based on this pre-established set of problem elements, and a procedure is described for development of an objective prioritization from the questionnaire data. The fourth section describes empirical re-

THE MULTI-ATTRIBUTE Often the use of a systematic approach in decision situations can enhance the effectiveness of the decision process (Sage, 1981). The use of multi-attribute utility theory attempts to enhance a decision process by explicitly defining the relative importance of the attributes of alternatives for an individual decision maker. This technique has produced favorable results in a wide variety of applications (Keeney & Raiffa, 1976). Consider the specific decision problem of designing an information system for use in the evaluation of police services. The problem of deciding what information to include in such a system must involve a determination of what are considered to be the most important objectives of police. One approach to this problem would be to simply poll samples of different categories of stakeholders within the Criminal Justice System (CJS) as to their ranking of the relative importance of a predetermined set of objectives. (Assume that this predetermined set of objectives would emcompass all major objectives.) There are at least two potential problems with this approach. The first is that direct assessment by stakeholders may fail to consider all of the important aspects of the problem (Sage, 1981). The second problem is that rankings provide no information on the relative differences in importance between objectives within the ranking. As an alternative to this approach, a systematic procedure could begin to address these two problems. This could be accomplished by determining the characteristics of information for evaluating police performance that make it useful, and the extent to which information on different objectives has these characteristics. If the relative importance of these characteristics of performance information can be determined, a systematic procedure could be developed for determining the relative importance of performance information on a set of objectives. In essence, information on objectives would be evaluated relative to its useful characteristics, and a ranking of the importance of information on objectives would follow from the relative importance of these characteristics. If applied in the context of a multi-attribute value function, such an approach can also provide a measure of the relative magnitude of importance of information on each objective. In a different context, Deutsch and Malmborg (1982a, 1982b, 1982c, 1982d) developed a model to evaluate organizational performance measures. This modeling approach is based on multi-attribute utility

sults contained in questionnaire responses obtained from six different stakeholder groups. Comparisons between, and within, stakeholder groups are provided and additional aggregate comparisons are made. The final section offers a summary and conclusions.

UTILITY APPROACH concepts involving determination of the relationship between organizational constituents, their activities, and the relationship between these activities and existing performance measures. The matrix model then develops a means to measure what activities individual constituents are involved in and to what extent. From this, a value function relating individual performance measures to various sets of organizational constituents is developed. Some of the concepts within the model of Deutsch and Malmborg (1982a, 1982b, 1982c, 1982d) can be adapted to the problem of evaluating the relative importance of information on police objectives. One way to do this is with the use of a simple additive value function. This value function is assumed to capture the extent to which information on the various objectives of police exhibits the desirable characteristics of information, and the relative importance of these characteristics. Thus, if m, represents a collection of specific information elements applicable to objective j and wi represents the relative importance of characteristics i, the value function for evaluating m, is of the form;

f(m,)

=

2 wlkij . r=l

In the above, ki, is a numerical value representing the relative extent to which information on objective j exhibits characteristic i, and n is the number of characteristics. In using such values as wi and kij in this way, we assume that individuals can accurately state how they utilize information and the relative importance of the characteristics of this information. This, however, seems reasonable for the present study since stakeholders within the criminal justice system presumably have as good an insight as anyone into the way they utilize information in evaluating police services. It should also be noted that use of the additive form off(mj) assumes that preferences associated with characteristics are mutually preferentially independent. This requires that the importance of one characteristic for any m,, is not affected by the levels of other characteristics achieved by that m,. If this assumption did not hold, an alternative form of f(m,) taking explicit account of interdependencies may be appropriate. However, such an alternative form of f(m,) would

Evaluating Police Services tend to require a detailed interactive session with a decision maker and thus be infeasible to implement for a sizable sample of stakeholders within the CJS. Thus, the use of the simple additive model makes access to a larger group of stakeholders feasible and facilitates the investigation into the hypothesis that different stakeholders have different perceptions of the importance of activities using thef(m,) values.

15

The set of objectives, characteristics, information elements, and wi and kij values are assessed by stakeholders within the CJS using the questionnaire described in the next section. In describing the assessment process, we denote the jrh objective as aj, and the i’h characteristic as ci. This notation is used throughout the remainder of the paper.

QUESTIONNAIRE DESIGN AND DATA DEVELOPMENT Given the additive form off(mj), it is possible to obtain the data necessary to evaluatef(m,) using a questionnaire. However, the design of such a questionnaire requires specification of a predetermined set of objectives, information sources associated with objectives (we will refer to these as “measures”), and the useful characteristics of performance information. These elements were drawn from a project on police performance measurement conducted by the American Justice Institute (Dunn, 1981). A total of six major objectives were specified and several measures were suggested as applying to each objective. In addition, sixteen useful characteristics of information were designated. The set of six major objectives include: (1) Crime Prevention (a,): This objective involves the minimization of the occurrence of personal and property crimes. (2) Crime Detection and Apprehension (a,): This objective involves the detection of offenses and the identification of offenders as well as their apprehension. (3) Maintenance of Social Order (a,): This objective involves the minimization of deaths, injuries, property damage, civil disturbances, and other criminal activity brought about by interpersonal conflict, intergroup conflict, and personal stress. (4) Traffic Control (a.): This objective involves maintenance of safety and efficiency in the movement of individuals and vehicles.

The set of 16 useful characteristics include:

c,: makes good use of financial and human police resources, cl: not overly technical in method of presentation, cJ: focuses on specific aspects of police work, cs: provides good statistical data, cs: provides a basis for planning the future activities of police, cs: increases the visibility of police, c,: is generally acceptable to all segments of the community, ce: is objective and unbiased, cs: contributes to the quality of life in today’s society, clO: gives practical information on how well policies or programs are operating, cI1: adds to general knowledge of police work, cL2: is inexpensive to gather, c,~: affects a high priority issue, ca4: increases the esteem in which police are held, cls: gives a comprehensive, overall view of the role of police, cL6: is understandable to everyone. Typical performance measures were associated with each objective that contained information on how well police were performing. These measures were obtained by asking various criminal justice practitioners what information was useful for them with respect to evaluating police performance. The measures suggested for each objective were:

l

(5) Emergency Services (as): This objective involves providing assistance to any person requesting it, or any person appearing in need of such assistance. (6) Community Relations (a,): This objective involves establishing and maintaining community esteem and support for the police by: maximizing the integrity of the police, providing information about crime to the community, and maintaining appropriate channels for feedback to the police from the community.

of information

l

Crime Prevention (a,): the total number (reported plus unreported) of personal crimes of homicide, rape, robbery, and aggravated assault; the number of actual property crimes of burglary, larceny, and vehicle theft; and the number of actual occurrences of all other personal and property crimes. Crime Detection and Apprehension (a,): the number of probable offenders identified for personal and property crimes, the number of personal and property crimes for which one or more actual offenders are apprehended, and the number of personal prop-

16

STUART JAY DEUTSCH

and CHARLES

erty crimes for which one or more actual offenders are convicted.

J. MALMBORG

cited, the number of traffic accidents resulting in property damage, and the number of traffic accidents resulting in death or serious injury.

Social Disorder (a,): the number of actual interventions by police in interpersonal and intergroup conflicts and personal stress situations, the number of interventions by police in interpersonal and intergroup conflicts and personal stress situations as a percentage of the total occurrences of these conflicts and situations, and the number of occurrences of death, injury, property damage, civil disturbances, and other criminal activity within a specified time period after intervention by the police.

Emergency Services (a,): the response time to emergency and priority calls, the number of incidents of emergency medical transportation provided by police, and the number of incidents of emergency first aid provided by the police. Community Relations (aa): the number of citizen complaints verified through judicial findings or internal investigation, the correspondence between police performance and goals of the community as evidenced through sample surveys and other means,

Traffic Control (a,): the number of traffic violators

COMPONENT FIVE: EMERGENCY SERVICES This component is based on the belief that a key function of the police in a community is to provlde assistance to any person requesting it or any person appearing in need of such assistance. Typical measures associated with this component include: (1) the response time to emergency priority calls, (2) the number of incidents of emergency medical transportation provided by police, and (3) the number of incidents of emergency first aid provided by police. To what extent is or does “Emergency Services”:

To a great extent

Largely

Somewhat

Slightly

Not at ail

Not applicable

(a) make good use of financial and human police resources?

1 1

1 I

[ 1

[ I

1 I

I 1

or presentation?

1 1

1 1

[ I

1 1

[ 1

[ 1

focus on specific aspects of police work?

1 1

[ 1

[ 1

I 1

[ 1

1 1

[ 1

1 1

1 I

1 1

[ 1

1 1

[ 1

1 1

[ 1

1 1

1 1

1 1

03 not overly technical in method (4

(4 provide good statistical

data?

63 provide a basis for planning the future activities

of police?

(9 increase the visibility of police?

i

1

1 1

1 1

[ I

[ 1

1 1

(9) generally acceptable to all segments of the community?

[ 1

1 1

1 I

1 I

[ 1

1 1

1 1

t I

1 1

1 1

1 I

[ 1

09

objective

and unbiased?

(9

contribute to the quality of life in today’s society?

[ 1

I I

1 1

[ 1

[ 1

[ I

(j)

give practical information on how well policies or programs are operating?

1 1

I 1

[ 1

1 I

[ I

1 1

(4

add to general knowledge police work?

1 1

[ 1

1 I

1 1

[ 1

I 1

1 1

1 1

1 1

[ 1

1 1

[ 1

(1) inexpensive

of

to gather?

(m) affect a high priority

issue?

1 I

1 1

1 1

[ 1

[ 1

I 1

increase the esteem in which police are held?

1 I

[ I

1 1

[ 1

1 1

[ 1

(0) give a comprehensive, overall view of the role of police?

1 1

[ I

1 1

I 1

1 1

[ I

(P) understandable society?

1 1

1 1

[ 1

1 1

[ 1

[ 1

(4

to everyone in Figure 1. Sample of the Questionnaire Pertaining to the Objective of Emergency Services (ua).

Evaluating Police Services and citizen satisfaction with police services as evidented through sample surveys. Value Function Assessment Using the Questionnaire Given the set of objectives, characteristics, and measures, the second major problem addressed in the design of the questionnaire was determination of a structure allowing evaluation of the k, and wi values used to define f(mj). To determine the k, values for a given objective, a,, respondents were asked to state the extent to which the corresponding measure set m, had each of the 16 characteristics. Respondents could reply

17

that the measure set m, had characteristics ci: to a great extent, largely, somewhat, slightly, not at all, or was not applicable. As an example, Figure 1 shows the page of the questionnaire corresponding to the objective of emergency services. At the top of the page shown in Figure 1, respondents were reminded of the corresponding measure set. Instructions were given to check the appropriate box for each characteristic listed along the vertical axis. To determine the relative importance of the sixteen characteristics, i.e., wl, . . . , wls, an additional page of the questionnaire shown in Figure 2 was used. Re-

IV. EVALUATlNG THE CHARACTERlSTICS OF INFORMATION: Next, we would like your assessment of how important certain types of information about police performance are to you. Below are listed the 16 characteristics of information you have just used to rate the approaches to police performance. When you are deciding whether or not information about police performance is useful, how important are each of these characteristics to you? Remember, we are interested in your feelings about these general characteristics, and not in how they may or may not relate to any of the approaches to police work discussed earlier. Please check the approprlate bracket next to each characteristic below. To what extent should information

about police performance

Essential

have the following

Highly Desirable

characteristics?

Desirable

Unnecessary

Undesirable

(4 make good use of financial and human police resources

(b)

not overly technical

1 1

t 1

1 1

1 1

[ 1

1 1

[ 1

1 1

1 1

I 1

[ 1

1 1

[ 1

[ 1

[ 1

in method

or presentation

(c) focus on specific aspects of police work

(d) provide good statistical

data

1 1

1 1

1 1

[ 1

1 1

(e) provide a basis for planning the future activities of police

I 1

I 1

I 1

1 1

I 1

(f)

[ 1

t 1

[ 1

1 1

1 1

1 1

[ 1

1 1

[ 1

I 1

[ 1

1 1

1 1

[ 1

1 1

[ 1

[ 1

1 1

1 1

1 1

1 1

[ 1

1 1

[ 1

1 1

[ 1

1 1

1 1

[ 1

1 1

(1) inexpensive to gather

[ 1

[ 1

1 1

[ 1

1 1

(ml affect a high priority issue

1 1

[ 1

1 1

1 1

[ 1

1 1

1 1

[ 1

1 1

[ 1

[ 1

1 1

[ 1

[ 1

1 1

[ 1

1 1

1 1

[ 1

1 1

increase the visibility

of police

(a) generally acceptable to all segments of the community

0-4 objective and unbiased 0) contribute

to the quality of life

in today’s society

(i) give practical information

on how well policies or programs

are operating

(k) add to general knowledge police work

of

ON increase the esteem in which police are held

(0) give a comprehensive,

overall

view of the role of police (Pj understandable society

to everyone in

Figure 2. SUII@ of Questionnaire

Pertaining to Assessment of the Relative lrnpotinee

of Cha~cteflstfcs.

STUART JAY DEUTSCH and CHARLES J. MALMBORG

18

NUMERICAL

Response for al with respect to c,:

TABLE 1 SCALE FOR k,, VALUES

To a Great Extent

Largely

Somewhat

Slightly

Not at All

1.0

9.75

0.5

0.25

0.0

Weight (b,,)

Not Applicable 0.0

TABLE 2 NUMERICAL

SCALE FOR w, VALUES

Response for q:

Essential

Highly Desirable

Desirable

Unnecessary

Undesirable

Weight (b,)

1.0

0.75

0.5

0.25

0.0

spondents could reply that individual characteristics were: essential, highly desirable, desirable, unnecessary, or undesirable for the purpose of evaluating police performance. To obtain assessments of k,, and wi for i = 1, . . . , 16andj = 1,. . . , 6, a numerical scale was associated with the questionnaire responses. In the case of the ki, values, the numerical scale was as shown in Table 1. Based on which box is checked by a respondent, the quantity bii assumes its values from the measurement in Table 1. Given the values of bi, for i = 1, . . . , 16 and j = 1, . , . , 6, the values of ki, are obtained by normalizing over the set of A objectives, i.e.,

k=1

fori = 1,. . . , 16andj = 1,. . . ,6.Inthecaseofthe wi values, the numerical scale associated with the questionnaire response is shown in Table 2. Based on a given response, 6, assumes a value from the measurement scale in Table 2. The value of wi is then obtained by normalizing over the set of characteristics, i.e., b/L6 wi

=

xbi r=l

16. Using values obtained in this way, fori= l,..., the evaluation of the measure set applying to objective j, i.e., m, was given by: 16 f(mj)

=

C

Wi

ktj

r=l

j=1,...,6. OVERVIEW

Measurement Considerations Given the form of the questionnaire and multiple respondents in stakeholder categories, numerous measurement related questions arise (Krantz, Lute, Suppes, & Trersky, 1971). Specific issues include the effects of cognitive biases in responses and the nature of respondent preferences (Sage, 1981). With respect to cognitive biases, respondents may have difficulty relating to the specific numerical measurement scale used in the evaluation. The scale used obviously affects the value of f(m,) and thus alfernative combinations of scale values should be examined to determine the sensitivity off(m,) to them. Also, different respondents within a stakeholder group may interpret the questions differently. Questions pertaining to preference structure actually go further than just the issue of preferential independence. More fundamentally, the assessment of f(m,) assumes that preferences are compensatory, i.e., a high or low value with respect to one characteristic can be offset by high or low values with respect to other characteristics. In the case of some respondents this assumption may be invalid and thus corresponding f(m,) estimates could be highly insensitive to extreme levels of a few key characteristics. Detailed investigation into these critical measurement issues is not feasible when a great deal of information is sought from a large number of respondents. Furthermore, use of a large sample of respondents will offset the impact of these factors (Edwards, 1975). In the following section, results using the measurement scales specified are presented and discussed. Applications of the matrix model where extensive consideration is given to theoretical and practical measurement issues can be found in (Deutsch & Malmborg, 1982a, 1982b, 1982~).

AND ANALYSIS

The study was implemented by distributing copies of the questionnaire form to five different groups of

OF RESULTS

stakeholders within the CJS of metropolitan Pittsburgh. These stakeholder groups included line police

19

Evaluating Police Services

SUMMARY

OF THE MEAN

TABLE 3 w, VALUES FOR EACH STAKEHOLDER

GROUP

Stakeholders

Characteristics CL c1 c. c4 cs CS c7 CS CS Cl0 Cl‘ cl, Cl, CM Cl, Cl6

Line Police

supervisory Police

District Attorneys

CJS Researchers

Citizens

0.056

0.057 0.039 0.051 0.069 0.067 0.075 0.060 0.066 0.072 0.057 0.057 0.075 0.042 0.064 0.046 0.063

0.097 0.066 0.046 0.067 0.092 0.055 0.057 0.042 0.050 0.034 0.042 0.067 0.036 0.059 0.064 0.060

0.092 0.066 0.049 0.056 0.066 0.073 0.067 0.063 0.053 0.039 0.039 0.063 0.049 0.076 0.056 0.063

0.076 0.062 0.053 0.068 0.067 0.061 0.067 0.053 0.046 0.049 0.053 0.075 0.044 0.061 0.066 0.057

0.059 0.053 0.062 0.077 0.056 0.066 0.065 0.065 0.053 0.065 0.066 0.050 0.067 0.046 0.071

officers, supervisory police officers, district attorneys, CJS researchers, and citizens. Each individual in each group represents a distinct evaluating constituency for which a separate value function is assessed. The stakeholder group referred to as citizens includes 29 volunteer participants from the Pittsburgh metropolitan area including students and selected residents from a mix of age and racial groups. The remainder of the total of 57 respondents includes 9 line police officers, 8 supervisory police officers, 6 assistant district attorneys, and 5 CJS researchers. In addition to individual results, aggregate results for stakeholder categories were obtained by averaging the responses, (i.e., the k, and wi values), of individual respondents in each group. Table 3 summarizes the mean values of wI through wX6for each stakeholder category. Table 4 summarizes the mean values off(m,) throughf(m,) for each stakeholder category where m, represents the measures associated with objective a, for j = 1, . . . , 6. To determine if respondents as a whole maintained different perceptions as to the importance of measures relating

MEAN AND STANDARD

Line Police Supervisory Police Dlstrlct Attorneys CJS Researchers Citizens

DEVIATION

0.156; 0.173; 0.190; 0.196; 0.176;

0.036 0.026 0.011 0.036 0.045

TABLE 4 OF Qm,) CORRESPONDING

0.201; 0.162; 0.203; 0.195; 0.174;

0.056 0.033 0.027 0.025 0.040

0.134; 0.157; 0.136; 0.143; 0.153;

m, =

measures applylng to objective e,, j = 1, . . . , 6 = value function mean corresponding to m,, j = 1, U,lrn,, = standard deviation of f(m,) r(m)

to different objectives, a one-way analysis of variance was implemented for the data in Table 4. In this analysis the hypothesis Ho: f(m,) = f(m,) for i,j = 1, . . . , 6, is tested. Thus, the simple statistical test attempts to determine if the relative evaluation of the importance on each objective as measured by the value function f(m,) is different across the objective set. Observations of f(m,) correspond to the mean value of f(m,) for each of the five stakeholder groups, for j = 1, . . . ,6. At the 99% confidence level, the null hypothesis is rejected, suggesting that stakeholders perceive differences in these values. Tests on individual means indicate that significant differences exist between the following pairs of meanf(m,) values: f(m,) andf(m,), f(M andf(mAf(mS andf(m3, andAm,) andf(m,). These results indicate that, overall, stakeholders consider measures relating to crime prevention, crime detection/apprehension, and emergency services as more important for evaluating police performance than measures relating to social order. In addition, measures relating to crime detection/apprehension are considered more important than measures relating to

...,6

TO EACH OBJECTIVE

0.065 0.027 0.029 0.046 0.046

0.162; 0.149; 0.156; 0.127; 0.153;

0.053 0.046 0.031 0.034 0.052

AND STAKEHOLDER

0.166; 0.171; 0.190; 0.156; 0.193;

0.044 0.024 0.036 0.029 0.043

GROUP

0.159; 0.166; 0.126; 0.163; 0.149;

0.041 0.043 0.035 0.027 0.056

STUART JAY DEUTSCH and CHARLES J. MALMBORG

20

traffic control. The standard deviations of the mean f(mj) values are also shown in Table 4 for each stakeholder group. To determine if the level of variability in stakeholder responses changes with measures relating to different objectives, a second analysis of variance was performed. In this case, the null hypothesis was given by N,,: arc_, = u,,,, for i,j = 1, . . . , 6, where uflm,, is the standard deviation of f(m,) obtained from a given stakeholder group for j = 1, . . . , 6. Observations of q,,, in the analysis of variance correspond to the u,(,~, values computed for each of the five stakeholder groups. At the 99070, 95070, and 90% confidence levels, the null hypothesis is accepted. This result suggests that the variability of responses within stakeholder groups does not change with the objective being considered. In the next section, the responses giving rise to the results in Table 4 are examined more closely within individual stakeholder groups. Analysis of Results Within Stakeholder Groups The results shown in Table 4 indicate that the order of importance (according to line police) of measures relating to objectives is given by: crime detection/apprehension (a,), traffic control (a,), emergency services (as), community relations (as), crime prevention (a,) and maintenance of social order (a,). The standard deviations of the f(m,) through f(m,) values obtained from the line police group are given in Table 5. From the data in Table 4, it is not immediately apparent whether or not the differences in the values of f(m,) through f(m,) for the line police are statistically significant. As a result, a one-way analysis of variance

TABLE 5

SUMMARY OF r(m,) MEAN VALUES AND STANDARD DEVIATIONS OBTAINED FROM LINE POLICE OFFICERS, AND ANOVA TABLE TO TEST SIGNIFICANCE OF DIFFERENCES BETWEEN r(m,) VALUES Mean Value Objective Crime Prevention (a,) Crime Detection/Apprehension Social Order (a,) Traffic Control (a,) Emergency Services (a,) Community Relations (a,) ANOVA

Total

Standard Deviation

0.156 0.201 0.134 0.182 0.166 0.159

0.036 0.058 0.065 0.053 0.043 0.041

TABLE FOR TESTING SIGNIFICANCE DIFFERENCES IN f(m,) VALUES

Source Objectives Error

(a,)

of f(m,)

f(m,)

Sum of Squares

Degrees of Freedom

Mean Square

.0264 .1142

5 48

.9953 .0024

.1406

53

OF

F, 2.23

was run for the line police respondents pothesis:

to test the hy-

Ho: f(m1) = f(m,) = f(m3) = f(m,) = f(m,) vs. H,: f(rni) * f(rnj) for at least one i * j: i, j = 1, . . . , 6. This test is aimed at determining if line police perceive significant differences in the relative importance of the objectives of police. Nine observations corresponding to individual line police respondents were available to test this hypothesis which was accepted at the 99% and 95% confidence levels and rejected at the 90% confidence level. The corresponding ANOVA table is shown in Table 5. The results in Table 5 indicate that line police officers do not perceive great differences between the importance of measures associated with the six major objectives in evaluating police performance. At the 90% confidence level, tests on individual means indicate that only the pairs: S(m,) and f(m,), f(m,) and f(m,), and f(m,) and f(m,) have significant statistical differences between them. This suggests (at the 90% confidence level) that line police consider crime detection/ apprehension and traffic control measures to be more important than measures of maintaining social order, and that measures of crime detection/apprehension are more important than measures of crime prevention for the purpose of evaluating their performance. One implication of this result is that line police officers may be more concerned with reacting to the occurrence of crime than preventing it, at least in as far as their performance evaluation is concerned. With respect to supervisory police, Table 4 indicates that the rank ordering of importance of measures related to the objectives is: crime detection/apprehension (a,), crime prevention (a,), emergency services (as), community relations (a,), social order (a,), and traffic control (a,). The standard deviations of the f(m,) through f(m,) values obtained from the line police group are given in Table 6. A one-way analysis of variance was implemented to determine if differences in the f(mj) values obtained from supervisory police were statistically significant. The analysis was based on eight observations representing individual supervisory police respondents. As the corresponding ANOVA table shown in Table 6 indicates, there were no significant differences between the f(mj) values at the 99070,95%, or 90% confidence levels. Thus, there is no evidence from thef(m,) values that supervisory police perceive any difference in the importance of measures relating to the six objectives for evaluating police performance. In addition, it is interesting to note that the standard deviations of the f(m,) values are lower in the case of supervisory police versus the analogous values for line police. This may

Evaluating Police Services TABLE6 SUMMARY OF r(m,) MEAN VALUES AND STANDARD DEVIATIONS OBTAINED FROM SUPERVISORY POLICE OFFICERS, AND ANOVA TABLE TO TEST SIGNIFICANCE OF DIFFERENCES BETWEEN f(m,) VALUES

Objective Crime Prevention (a,) Crime Detection/Apprehension Social Order (a,) Traffic Control (a,) Emergency Services (e,) Communlty Relations (a$

(a,)

Mean Value of f(Q)

Standard Deviation

0.173 0.162 0.157 0.149 0.171 0.166

.026 .033 .027 .046 .024 .043

ANOVA TABLE FOR TESTING SIGNIFICANCE OF DIFFERENCES IN f(m,) VALUES

Source

Sum of Squares

Degrees of Freedom

Objectives f(m,) Error

a049 .0502

5 42

.0551

47

Total

Mean Square .OOlO .0012

F, 0.6367

suggest, for example, that supervisory police are a more homogeneous group than line police in their perception of the relative importance of the objectives of police. In the case of district attorneys, the order of the f(m,) throughf(m,) values corresponded to the ordering of objectives given by: crime detection/apprehension (a2), emergency services (as), crime prevention (al), traffic control (a,), social order (a,), and community relations (as). The standard deviations of the f(m,) through f(m,) values obtained from the district attorneys group and the ANOVA table for testing for differences in thef(m,) values are shown in Table 7. The one-way analysis of variance indicates that there were statistically significant differences between the f(mj) values at the 99% confidence level. Tests on individual means reveal that there were significant differences between the following pairs of values: f(m,) andf(Q,f(m,) andf(m&f(m,) andf(W, f(W and f(m3,f(m2) andf(ms),f(m3) andf(m,), and.!+@ and f(ms). These results imply that district attorneys consider measures of social order to be significantly less important than measures of crime prevention, crime detection/apprehension, and emergency services. Measures of community relations are considered to be significantly less important than measures of crime prevention, crime detection/apprehension, and emergency services. Finally, measures of crime detection/ apprehension are considered by district attorneys to be more important than measures of traffic control. These results are particularly interesting in light of the fact that district attorneys as a stakeholder group have the

21

lowest average standard deviation of thef(m,) values. In a sense, the results for district attorneys are not surprising since the efforts of police in crime detection/ apprehension have a direct impact on their ability to prosecute criminals and therefore they consider related measures as most important for evaluating police. In the case of CJS researchers, the order of thef(m,) through f(m6) values corresponded to the ordering of objectives given by: crime prevention (a,), crime detection/apprehension (a,), community relations (as), emergency services (as), social order (a,), and traffic control (a& The standard deviations off(m,) through f(m,) and ANOVA table for the CJS researcher stakeholder group is given in Table 8. The hypothesis of equal f(mj) values (i.e., Ho) for the CJS researcher stakeholder group was accepted at the 99% confidence level and rejected at the 95% confidence level. Tests on individual means at the 95% confidence level revealed significant differences between the pairs of f(mj) values given by: f(m,) and f(m.4, f(m,) and f(m3, f(m,) and f(U f(m,) and f(m,), and f(m,) and f(m,). This suggests that CJS researchers consider measures of crime prevention and crime detection/apprehension to be more important than measures of social order and traffic control. In addition, measures of community relations are considered more important than measures of traffic control for evaluating police performance. As a stakeholder group, CJS researchers had the second-lowest average standard deviation of the f(mj) values. The results for CJS researchers suggest an emphasis on measures relating to the level of reported offenses. This may suggest a quantitative orientation in evalu-

TABLE7 SUMMARY OF f(q) MEAN VALUES AND STANDARD DEVIATIONS OBTAINED FROM DISTRICT ATTORNEYS, AND ANOVA TABLE TO TEST SIGNIFICANCE OF DIFFERENCES BETWEEN f(m,) VALUES

Objective Crime Prevention (a,) Crime DetectionlApprehenslon Social Order (a,) Traffic Control (a.) Emergency Servlces (a,) Community Relations (ad

Mean Value of wn,)

Standard Devlatlon

0.190 0.203 0.136 0.156 0.190 0.126

0.011 0.027 0.029 0.031 0.036 0.035

(aI)

ANOVA TABLE FOR TESTING SIGNIFICANCE OF .DIFFERENCES IN f(q) VALUES

Source

Sum of Squares

Degrees of Freedom

Mean Square

Objectives f(m,) Error

.0317 .0246

5 30

.0063 .ooO6

.0562

35

Total

F, 7.66

STUART JAY DEUTSCH and CHARLES J. MALMBORG

22

TABLE9 SUMMARY OF f(q) MEAN VALUES AND STANDARD DEVIATIONS OBTAINED FROM CITIZENS, AND ANOVA TABLE TO TEST SIGNIFICANCE OF DIFFERENCES BETWEEN f(q) VALUES

TABLE8 SUMMARY OF f(m,) MEAN VALUES AND STANDARD DEVIATIONS OBTAINED FROM CJS RESEARCHERS, AND ANOVA TABLE TO TEST SIGNIFICANCE OF DIFFERENCES BETWEEN f(m,) VALUES

Objective Crime Prevention (a,) Crime Detection/Apprehension Social Order (a,) Traffic Control (a,) Emergency Services (as) Community Relations (a.)

Mean Value of f(m,)

Standard Deviation

0.196 0.195 0.143 0.127 0.156 0.183

0.038 0.025 0.048 0.034 0.029 0.027

(a,)

Objective Crime Prevention (a,) Crime Detection/Apprehension Social Order (a,) Traffic Control (a.) Emergency Services (a,) Community Relations (a,)

Objectives Error Total

Sum of Squares

Degrees of Freedom

Mean Square

.0212 .0293

4 25

.0042 .0012

.0505

29

f(m,)

F,

3.44

of Results Between Stakeholder

It is of interest to maintain the same measures relating to to this would be to

Source Objectives Error Total

ating police performance on the part of this stakeholder group, with a focus toward serious, as opposed to minor, offenses. The results obtained from 29 citizen respondents are summarized in Table 9. Based on the mean values of f(m,) throughf(Q, the ordering of objectives for this stakeholder group is given as follows: emergency services (as), crime prevention (a,), crime detection/apprehension (a,), traffic control (a,), social order (a,), and community relations (a&. The one-way analysis of variance rejected the hypothesis that the values off(m) throughf(m,) were equal at the 99% confidence level. Tests on individual means indicated that pairs of f(mi) values with significant differences between them included: f(m,) and f(m,), f(m,) andf(mA andf(mJ andfhd. These results suggest that citizens consider measures of emergency services to be more important for evaluating police performance than measures of traffic control, social order, and community relations. One possible implication is that citizens perceive the role of police as providers of emergency services to be as important at their role in law enforcement. In addition, measures relating to traffic control and community relations (two areas where many citizens have their only contact with police) are perceived as less important in evaluating police performance. Analysis

Standard Deviation

0.178 0.174 0.153 0.153 0.193 0.149

0.045 0.040 0.048 0.052 0.043 0.058

(a,)

ANOVA TABLE FOR TESTING SIGNIFICANCE OF DIFFERENCES IN f(m,) VALUES

ANOVA TABLE FOR TESTING SIGNIFICANCE OF DIFFERENCES IN f(m,) VALUES

Source

Mean Value of W,)

Groups

determine if stakeholder groups perceptions of the importance of different objectives. One approach test the hypothesis that the values

f(m,)

Sum of Squares

Degrees of Freedom

Mean Square

.0460 .3649

5 168

.0092 .0022

.4109

175

F,

4.24

of f(m,) through f(m,) assessed for different stakeholders have no significant differences between them. For example, we can test if the values off@,) assessed for line police, supervisory police, district attorneys, CJS researchers, and citizens were significantly different. If not, this would indicate that the stakeholders maintain similar preferences with respect to the importance of measures of crime prevention relative to measures applying to different objectives. Conversely, if statistical significant differences were found, this would suggest that different stakeholder groups may have different preferences on the relative importance of measures relating to crime prevention. Thus in the case of m, we test the null hypothesis given by: Ho: fl(mj) f’(mj) = f2(m,)

=

f”(mj)

=

f*(mj)

=

f”(m,)

=

= f(mJ = f”(mJ = f’(mJ = f”(m,), where meanf(mj) value assessed from line police, mean f(mj) value assessed from supervisory police, mean f(mj) value assessed from district attorneys, mean f(mj) value assessed from CJS researchers, and meanf(mj) value assessed from citizens.

In order to test this hypothesis, an unbalanced analysis of variance is performed since sample sizes in each stakeholder group are unequal, and observations on the mean values correspond to f(m,) values obtained from individual respondents in stakeholder groups. The analysis was performed for each of the six objectives.

Evaluating TABLE 10 F, VALUES FROM TESTING H,: f’(m,) = fyn?,f = f”(m,) = fvb) = f”(m,) FOR EACH OBJECTIVE (52 ERROR DEGREES OF FREEDOM)

Objective

F0

Crime Prevention Crlme Detection/Apprehension Social Order Traffic Control Emergency Services Community Relations

1.1233 1.4806 0.4424 1.4737 1.8083 1.0741

Table 10 shows the test statistics obtained from the analysis of variance applied to each objective. In the case of each objective, the null hypothesis was accepted at the 99070, 95Q70,and 90% confidence levels. These results indicate that different stakeholder groups have roughly equivalent perceptions of the relative importance of measures relating to the six objectives. This is true even though previous results indicate that stakeholders do differentiate between the relative importance of measures relating to the six objectives. One possible implication of this result for the design of an information system for evaluating police

Police Services

23

performance is that measures considered to be relatively important by one stakeholder group are apt to be considered relatively important by other stakeholder groups as well. Up to this point, the results from this study have suggested that different stakeholder groups have similar preferences for the importance of measures relating to different objectives. In addition, it has been found that average standard deviations of the f(mj) values are roughly the same for the six different objectives. However, it is not clear whether the same degree of variability exists within different stakeholder groups. To investigate this possibility, we can test for signific~t differences in the size of the average aft,li, values obtained from different stakeholder groups. These tests were performed and revealed that significant differences existed between the standard deviations of line police and district attorneys and between district attorneys and citizens. In particular, the average standard deviations of citizens and line police were found to be significantly higher than those of district attorneys. This result suggests that district attorneys have more uniform preferences than line police as to the importance of measures relating to different objectives.

SUMMARY AND CONCLUSIONS This paper has reviewed a study to determine if different stakeholders within the CJS attach the same importance to measures used in evaluating police performance that apply to different objectives. Implementation of the study was accomplished through the use of a questionnaire dist~buted to 57 respondents representing five different stakeholder groups within the CJS of the Pittsburgh metropolitan area. The questionnaire was designed to allow the evaluation of a simple additive model based on the matrix model of Deutsch and Malmborg (1982a, 1982b, 1982~). This value function was used to evaluate the relative importance of prespecified measure sets for use in evaluating police, and was based on a set of six different, prespecified objectives. Use of an additive model and the questionnaire format raised several measurement issues relating to the preference structures of respondents and cognitive biases that warrant further investigation.

The results of the study implied that the variability of preferences for different types of information was greater among line police and citizens than other stakeholder groups. OveraIl, stakeholders did perceive significant differences in the importance of different measure sets with the greatest differences being noted by district attorneys and citizens. Perhaps most significantly, the analysis found strong consistency in the relative importance of measures expressed by different stakeholder groups. In particular, no differences were detected among stakeholders relative to any individual measure set. The implication of this finding for the design of an information system used in the evaluation of police performance is that the same basic information content can potentially serve several different evaluating constituencies.

REFERENCES CONNOLLY, T., CONLON, E. J., & DEUTSCH, S. J. (1980). Grg~tio~ effectiveness: A multipi~~o~tituency approach. The Academy of Management Review, 5, 1.

DEUTSCH, S. J., &cMALMBGRG, C. J. (1982b). The design of organixationai performance measures for human d~~ionm~ing, Part II: implementation example. IEEE Transacfion on Systems, Man, and Cybernetics, SMC-12(3), 353-360.

DEUTSCH, S. J., & MALMBORG, C. J. (198Za). The design of organixational performance measures for human d~isio~aking, Part 1: Description of the design methodology. IEEE Tmnsactiom on Systems, Man, and Cybernetic, SMC-B(3), 344-353.

DEUTSCH, S. J.. % MALMBORG, C. J. (1982~). The evaluation of performance measures in the presence of interdependent organizationd activities and components. ZEEE Tmrwctions on Systems, Man, and Cybernetics, SMC-12(3), 361-370.

24

STUART JAY DEUTSCH and CHARLES

DEUTSCH, S. J. & MALMBORG, C. J. (1982d). The utilization of prevailing evaluation policy in the selection of performance measures, Evaluation and Program Planning, 5, 149-159. DUNN, W. (1981). Pittsburgh: Graduate School of Public and International Affairs, University of Pittsburgh. Personal communication. EDWARDS, W. (1975). Use of multiattribute utility measurement for social decision-making. In D. Bell, R. L. Keeney, & H. Raiffa (Eds.), Conflicting objectives in decisions. New York: Wiley.

J. MALMBORG

KEENEY, R. L., & RAIFFA, H. (1976). Decisions with multiple objectives: Preferences and value tradeoffs. New York: Wiley. KRANTZ, D. H., LUCE, R. D., SUPPES, P., & TRERSKY, A. (1971). Foundations of Measurement, Volume I: Additive andpolynomial representations. New York: Academic Press. SAGE, A. P. (1981). Behavioral and organizational considerations in the design of information processes for planning and decision support. IEEE Transactions on Systems, Man, and Cybernefics, SMC-II, 5, W-678.