Evaluating the performance capability of a multi-organization system

O.VIEG.4 The [nt Jl or" Mgmt Sci, Vol. tl. N o 5, pp. 417-432. 19S3

')305-0483 S 3 $ ; 0 0 - / ) t ) 0 Cop}r~gh~ ~ 1983 Pergamon Press Ltd

Pr'mt~d {n Grea: Britain All ngh:s resewed

Evaluating the Performance Capability of a Multi-organization System t ISRAEL MOSHE

SPIEGLER BEN-BASSAT

Tel Aviv University, Israel YOSEF

DRAIZIN

YUVAL

KARMON

Mekorot, Israel Water Company (Received October 1982; in recised form January I983) A model for evaluating the performance capability of a multi-organization system is introduced. The model is based on hierarchical-modular decomposition of the components which determine system performance up to the level where they can be measured quantitatively, or at least qualitatively, in an objective or subjective manner. Performance measures are obtained by incorporating these components into multi-attribute evaluation functions. The model provides decision makers with a tool for quantitative assessment of performance, identification of bottlenecks, a framework for 'what-if' simulation and a mean for conflict resolution during planning. Application of the model to the Israeli Water Resources Development System provides an illustrative example.

1. I N T R O D U C T I O N " W E ARE now working at 703/0 (or any other number) of our full capacity" is a frequently heard statement in management meetings. Such a statement typically implies that, given the knowledge, facilities, market potential and other internal resources available to the enterprise, and considering external constraints, different operational procedures and organizational structures may improve performance. When the speaker tries to pinpoint the specific factors which, in his opinion, impede the company's success, a heated and excited discussion is triggered in which each executive protects his own department and passes the blame on to other internal or external factors. T h i s r e s e a r c h w a s s u p p o r t e d in p a r t b y M e k o r o t . Israel Water Company. at7

Interestingly enough, rarely is there an argument about the basic claim which stimulated the heated discussion; that the company is performing below its capacity. While the above statement, intuitive as it may be, is usually made because it is simply popular to be dissatisfied with present performance, management, as a result, is faced with a twofold problem: to assess the actual performance and to devise a plan of action for improving on it. Measuring the performance of an organization or the degree to which it utilizes its capacity is a difficult task, but may be necessary to substantiate the claim regarding the low performance. Even when a quantitative measure of performance is obtained, decision makers still differ as to the normative measures to be taken so that performance is improved. This paper treats these two aspects of organizational performance evaluation. It describes a model which provides

418

Spiegler

et

al.--Et'aluatin~ Perl-orrnance Capability

management with an effective method to measure the validity of the above statement, trace its sources and test by simulation the effect of changes in operation or structure on the system performance. As such it can expand the level of agreement among managers so that they focus their discussion on the real differences. We do not claim that our model offers an unequivocal answer to these seemingly endless discussions. However, it may elevate the level of debate, structure the discussion to an agreeable factoriented framework and direct it into more productive goal-oriented channels. It may thus contribute to conflict resolution by defining the precise nature of opinion differences. The model was developed for the Israel Water Resources Development System (IWRDS) and implemented successfully on their data. Still the model is general and may be adapted to other organizations. IWRDS is a multi-organizational system which initiates, designs, develops and implements facilities for supplying water to the state. The system consists of several autonomous organizations each responsible for some aspect of the overall objective. Among the organizations making up IWRDS are: water resources authority, the water company, water planning company, state supervisory unit, subsidiaries, subcontractors and others. IWRDS is a non-profit entity whose primary objective is to maximize the work completed within a given predetermined budget and program plan. A program plan consists of hundreds of projects each of which may require the involvement of several organizational units. Executing the annual plan requires concerted efforts of every organization within IWRDS. The various activities, such as detailed planning, equipment purchasing and real estate preparation, need to be scheduled and synchronized with resources allocated accordingly. Beyond the design of the critical path, the relative involvement of each organization in the various activities especially those which slow down the system performance need to be analyzed. The model described in this paper offers such an analysis. We took a top-down approach to the IWRDS study aiming to identify the factors which affect the level of the system's performance capability and to integrate these factors into a model which provides the following:

(1)

quantitative measures to assess the performance level on each of the system activities:

(2)

tools to identify bottlenecks and the organizational units responsible for them:

(3)

a framework for "what-if" simulation which permits the measurement of changes in the system performance as a result of changes in the operational strategy or structure.

The model is based on a hierarchicalmodular decomposition of the various components which determine system performance up to the level where they can be measured quantitatively, or at least qualitatively, in an objective or subjective manner. The components include the activities which are to be performed under the annual plan and the various organizational units and departments which carry out these activities. The activities are further decomposed in subcomponents which are called parameters. The parameters are associated with each of the activities through weights that represent their relative contribution to the performance of the respective activities and through evaluation functions which characterize the effect of changes in the parameters on the performance of the activities. The performance measures are based on multi-attribute evaluation (MAE) models which have been extensively explored in the past, both theoretically and in practice. Survey papers dedicated to this topic are [2] and [3]. Keeny & Raiffa [4] wrote a comprehensive textbook which covers this topic thoroughly. Edwards [1] offers guidelines for applying MAE models systematically. All of these publications also include discussions of the critical issues involved in the application of MAE technology. Section 2 presents an overview of the model. Sections 3 to 4 detail the evaluation functions and performance measures. Section 5 discusses the procedure used to assess model parameters. Model implementation is discussed in Section 6. The use of the model as a decision aid in evaluating planning policies, what-if situations and conflict resolutions are discussed and illustrated in Section 7. Results of experiments are shown in Section 8, followed by the summary and conclusions of the IWRDS study.

Omega. Vot. I I, ,Vo. 5 T~,I3LE [.

419

STa~TtONS a.ND t H E I R a,CT"D,',TIES

Sta~.ions k E" 5

_

2;

--

=

ActNities t

tll

~2)

(3)

(4)

(5)

(I) (2) 13) (4) (5) (6) 17) {81 I9) {10) (1 I ) (12) t 13) (14)

20 5 70 50

20 50 20 50

10 15 l0

50 20

l0

10 31 50 65 10 33 90 20

S0 t9 20 25 45

Apprai>al of o~erail plan Preparing development plans Preparing budget Ratif?ing development plans Detailed planning Purchasing ElectriciD orders Real estate preparations Contract> preparation Equipment supplied Electricit? supplied Pre-construction organization Construction Operation

2. M O D E L OVERVIEW The first step in constructing the model for IWRDS was to identify and specify the various acticities which cover the life cycle of the annual program plan. These activities range from the approval of the overall plan, through the preparation of detailed plans and budget and on up to construction and operation. The complete list of activities is shown in the rows of Table 1. Next to be specified were the various organizations and departments involved in performing the system activities. These are called stations in our model and are shown in the columns of Table 1. The 15 stations are grouped as follows: station 1 represents the organization responsible for budget allocation: stations 2 to 3 represent management; stations 4 to 5 represent the designing units; stations 6 to 7 represent the internal control units; stations 8 to 12 represent the subsidiaries and subcontractors responsible for the execution of the system projects: and stations 14 to 15 represent the suppliers of materials and equipment. Activities may be performed by one or several stations, each exercising different responsibilities over the activities involved. For example, activity 3, "preparing the budget', is o/ divided among water resources authority (70/°), management of Mekorot (the water company) (20°;) and PMU (the Projects Monitoring Unit)

-

-.,

.=

=

2

-- ~'

161

{7t

'(1

!%

~

IlO}

~

Illl

(12

113~ tlaj

1151

lO 3l I0

5 10 /4

6

13

20

14

15 2

2

i2

I2

5

33 I0

[0

10 20

5

40 20 a0

13 17 i5

2 3 3

11 i5 I0

II 15 10

3 5 2

(10%). The (i,k) entry of Table l - - t o be denoted a;k--indicates the relative involvement of station k in performing activity i. An empty entry represents no involvement of station k in activity i. The performance level of each activity, as contributing to the overall company performance, is a multi-dimensional function of several factors such as budget, time, manpower, complexity, equipment, managerial ability and changes introduced while work is in progress. It should be noted that we do not attempt to measure performance of activities as independent entities, but rather their performance as reflecting upon the overall system performance. This is a key issue since optimizing factors which affect positively the performance of an individual activity may have a negative effect on the overall utilization of the system capacity. For instance, the more time devoted to detailed design of projects, the better the designs; however, a delay in submitting the designs affects negatively' the utilization of overall system capacity, since at the end of the year the designs will look beautiful on paper, but no work will have been done in the field. The factors affecting the performance level of the activities are decomposed into subfactors up to the point where they can be measured in an objective or subjective manner. For example, activity 2 (preparing development plans) con-

Spie~.ler et aI.--Ecaluating Perjormance Capability

420

Ta.I3L,E 2.

TH~

-%CTI'~ITIES ~.ND RELATED Pa, RAMETERS

Activities i -=

-__

to

.a

=_..

Parameters j ,Money Time

Manpower l Complexity

t

Equipment t Management t Changes L

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (22) (23) (24)

Budget size End date 80"~ plans Termination date of activity i Date of equipment orders Work spread over year Schedule o~errun (months) Manpov,.er (subsidiary SHM) Manpower (subsidiary BP) Manpov, er--planning Manpower--PMU Manpower--supervision General complexity Number of facilities Number of contracts Geographical location Equipment ratio (to budget) Inventory ratio (to equipment) Special equipment Managerial ability--PMU Managerial ability--subsidiaries Contractors subsidiaries ratio Changes in work plan Changes in plans Changes in plan data

(I)

7

"z

= 7~

>, .~

(4)

(5)

(6)

(7)

30

10 14

5 12

5

---

--. =

(2)

(3) 30 30

30

44

~.

(81

-~.~_ E

20

5

20

:~ '~ = ~

10 21 I6

8

8 33 10

7 7 19 10 7

14 7 7 4 10

12 44

9 12

l0 15

3 8

5 I0

4

7 3

7 40

16 31

tributes to the overall performance through time and complexity. In this case the only relevant parameter of time is termination date ( j = 3). This is a quantifiable element and is measured directly in an objective manner. Complexity, on the other hand, is better measured when further decomposed into general complexity ( j = 12) and complexity resulting from the total number of facilities to be constructed during the development period ( j = 13). General complexity, such as technological constraints, is measured quantitatively as the percentage of nonstandard facilities in the plan; 'non-standard' being judged subjectively. Total number of facilities to be constructed is measured quantitatively in an objective manner. The factors and subfactors at the lowest level are called parameters and are listed in the rows of Table 2. An evaluation function V¢(x) is associated with each pair of activity i and parameter). The value of the function represents the desirability of having parameter j at value x for activity i. The evaluation functions for scoring individual parameters are described in the following sec-

40

10 10 3

10 10

7 II II

25 8 10

10 8 I2 8 5 5 5

10

17 17

11 16 16 10

60

=

(9) (10) (It) (12) (13) (14) 7 21

20

E

7 7 II 4

5 10 4 5 4 5 5 3 2 7 10 10

14 3 6 6 3 3 6 3 6 6 6 3 3 3 3 6 6

25 I0 15 10 10 15 15

6

tion. A weighted sum of these functions over the parameters which affect activity i provides a performance measure for activity i. The (i,j) entry of Table 2 - - t o be denoted wis--indicates the relative weight of p a r a m e t e r ) in performing activity i. Where no value appears, a zero weight is assumed. For instance, the score on performing activity 2 (preparing development plans) is a weighted average of the scores on three parameters: termination date, 44~ weight; general complexity, 12~ weight; and number of facilities, 44~o weight. For compact presentation we have chosen not to describe further the parameters in Table 2. However, their description is given unequivocally in the user questionnaire (Appendix) which serves as the input to the model. A weighted sum of the performance levels over all activities measures the system's ability to complete its work plan. These weights are shown in Table 3. We also use weights to describe the relative involvement of each organizational unit in the performance of each activity. By aggregating the performance level of activities related to a given organizational

Ome,ga, I,'ol. 1 1, 3;0. 5 T~BLE 3. ~EL~,I-I'~E tMPORr4NCE OF ~,(~I~IrIES Activit?

Weight

I. 2. 3. 4. 5 6. 7. 8. 9. 10. II. 12. I3. 14

6 3 3 6 15 10 6 6 12 6 6 6 t2 3 (100)

Appraisal of overall plans Prepanng development plans Preparing budget Ratif?ing development plans Detailed planning Purchasing Electricity orders Real estate preparations Contracts preparation Equipment supplied Electricity supplied Pre-construction organization Construction Operation (Totat)

unit we obtain measures on the unit's performance. An overall performance measure is finally derived by aggregating the performance on each activity weighted by its importance to the overall system performance. Table 4 summarizes the model components and the notation which will be used henceforth. Section 5--in which we describe the techniques for assessing these components--may shed further light on their definition and meaning.

3. E V A L U A T I O N F U N C T I O N S The following families of the evaluation functions, Q ( x ) , were identified for our application. For each of these functions the horizontal axis provides a scale for measurement of the relevant parameter. The vertical axis measures the performance score on a scale of 0 to 100. The shapes of these functions are depicted in Fig. 1. (1) Budget (parameter 1) The scale for this parameter is the ratio of approved to requested budget. The function is

TABLE 4. A SUMMARY OF MODEL COMPONENTS AND NOTATION

k j i %

= ~ = =

stations index. parameters index. activities index. relative importance of parameter j to the performance of activity i: ~ % = I for every i.

% ~ relative involvement of station k in the performance of activity i: ~ a~ = 1 for every, i. t, = relative importance of activity i to the overall performance of the system: S" t, = I. c,, = score of parameter j on activity i for a given value of parameter j.

-12[

a concave increasing function of the form I Fig. la): l,i.v ~ = (a - he

' ~ * 100.

{1)

It attains a zero value at a certain threshold limit Xmin which indicates a budgetary constraint (in percentage of requested budget) below' which the system will not be able to stand by its stated program. The upper limit "Xm~, is a 100°,; approved budget. In between, the larger the percentage ratio of approval, the higher the performance score. The rate of increase in performance is high in the neighborhood of,fm~ and it gets lower as we approach X~,,~. The parameters a, b and c determine the zero crossing point, i'm,,, the 100 score point, Jt'm~, and the concavity of the function. For example, budget size affects activities 3 and 4 ( j = I, i = 3 and i = 4) via function (1) such that when 50°/~ or less of requested budget is approved V,j(.v)= 0; and when 100°,; requested budget is approved the score is 100. The value of a, b and c in this case are 2.54, 4.18 and 1, respectively. (2) Time (parameters 2 to 4) These parameters measure the importance of performing the activities up to a given date. The x-axis represents months ranging from - 2 4 to + 2 4 where x = 0 is the beginning of the fiscal year. The performance score is a mirror image, decreasing S-shaped function of the form (Fig. lb): V(x)=

1

1 1-.-ae-~"

* 100.

(2)

See also [5] for an application of this function in marketing. This function is asymptotic to the left approaching the 100 score which indicates that there is a limit to gains from early completion of an activity. As long as the activity is completed prior to a specific date the score is practically 100. The mirror image S shape indicates a slow decrease in performance for little delay, a sharp decrease beyond a certain point and finally, close to the zero value, a moderate decrease which means that we are so late in the schedule that not much is left to be lost (i.e. beyond t months of delay it does not really matter any more if this delay is extended further). The function is asymptotic to the right approaching a zero value. The steepness of the

Spie~ler e t a .--Ecaluating Perjbrrnance Capabihty

422

(a)

(b)

/,

I00 80 "~ 60 40

20

50

t00 80

40 20-

iO0

Time (1"=2)

requested

(c)

I00

I 24 months

0

9 End p l a n s

(d)

I00

80

80

~" 6 0

60

40

40

20,

20 6 Schedule overrun (j-6}

-9

-24

approve~ (%)

Budget (j" I )

h

60

12 months

50 Manpower ( j = iO )

IO0

15o

-existing -(%) desirable

(e)

I00

(f)

ioo

80

80

"~ 60

60

40

40

20

20 300 Complexity (j-13)

600

900 Number

1200

1500

30

of f a c i l i t i e s

Flo. 1.

m

50 60

Equipment (j=16)

I00 equipment budget

Evaluationfunctions.

function and the shift right or left are determined by the values of a and b. For example, performance of activity 5 (detailed design) is given the score 100 when termination date of design ( j = 2) is nine months or earlier before the beginning of the fiscal year. A score near zero is given if the termination date is nine months or later into the fiscal year, since without a detailed design at this point in time there is no way the work plan can be completed this year. In this case, the values of a and b are 5 and 0.50, respectively.

where m h represents the proportional amount of work at month h relative to the overall amount. This function attains its m a x i m u m (100 score) for a uniform spread, i.e. m h = l / 1 2 for i = 1,2 . . . . . 12, and attains its minimum (0 score) when the entire a m o u n t of work is concentrated in one month, i.e. m t = 1 for a certain t and rn h = 0 for h # t. In between, the wider the spread over the year, the larger the value for V.

(3) Work spread over year (parameter 5)

(4) Schedule overrun (parameter 6)

This parameter is measured by the entropy

function: - t00,

m,

The scale for schedule overrun is in months

Omega, Vol. I1, .Vo. 5

over approved period. The evaluation function is (Fig. lc): I,'(x) = a -- bx:

14)

where a = 100 and b < 0. The function takes a concave decreasing shape giving a score o f 100 when no overrun has taken place. The score decreases as x increases to a value of 12 m o n t h s where the score is V(x)= 0. The function remains at the zero level for x > 12. F o r parameter j = 6 and activity i = 1 3 , a=100 and b = -0.69. (5) Manpower (parameter 7 to 11) In general, these parameters are linear or piecewise linear functions o f the form (Fig. ld): V(x) = ax + b

(5)

where the scale of x is a ratio o f existing manpower x= * 100, desired manpower o/ x m a y range from 0 to 200/0 . F o r example, parameter 10 ( P M U m a n p o w e r ) affects activity 5(detailed plan) in the following manner: it increases linearly up to the point where the ratio is 100% and then decreases linearly for a ratio larger than 100% up to 2000/o; that is, when the existing size o f m a n p o w e r is larger than desirable, performance decreases. In this example the values of a and b are: for x < = 100, a = 1.19, b = - 1 9 ; f o r x > 100. a = - 0 . 3 ,

b = 130.

(6) Complexity (parameters 12 to 15) These parameters measure the effect o f complexity factors, such as technological constraints, n u m b e r of facilities, n u m b e r o f contracts and geographical location o f majority of the projects, on the level of p e r f o r m a n c e capability. The evaluation functions are linear or piecewise linear. For instance, overall complexity ( j = 12) is a decreasing linear function with a = - 1, b = 100 measuring the score o f v(x) = 100 at x = 0. The function describing parameter 13 (number o f facilities) is made up o f three lines. For x between 50 and 200 facilities, the function increases from a score o f 70 to 100, respectively (a -- 0.2, b = 60). Between 200 and 500 the line is horizontal ~ving V(x) = 100, (a = 0, b = 100), and for x greater than 500, the function decreases linearly with a = - 0 . 1 and

-1.23

b = 150. Thus, 1500 facilities in the work plan yield a score o f 0 (see Fig. le). A similar function describes parameter 14 (number of contracts). Here, the x-axis is double the scale o f the previous parameter as on the average there are two contracts for each facility in the development projects. G e o g r a p h i c a l location, measured by the site o f 50,°,o o f the projects, also affects the performance of the system since it is much easier to work in the desert than in the m o u n t a i n s or in populated areas. Thus, a decreasing linear function measures the performance of this parameter, attaining a score of 100 when 50~o of projects are performed in the Negev (desert) and a score near zero when 50°0 or more of the projects are to be performed in the populated coastal area. (7) Equipment andsupp/ies (parameters 16 to I8) These parameters are measured in a linear or piecewise linear form. The x-axis is a percentage ratio of the equipment cost to total budget. For example, parameter 16 (equipment size) is given a 100 score for up to 3 0 ~ ratio. Between 30 and 60% ratio, the function decreases linearly to a score of 50 (a = - 1.67 and b = 150) and for x above 60°o the function is a horizontal line giving a score o f 50 (see Fig. l f). (8) Managerial abilities (parameters 19 to 21) The score for these parameters is estimated directly in the form: V~x) = x.

16)

(9) Changes (parameters 22 to 24) Similar to the budget parameters discussed above, these parameters are measured with the function: V(X) = (a -- be -'~) * 100.

(7)

For example, parameter 22 (changes in work plan) is given a score o f 0 when there is a 50~'Jl; or less ratio o f facilities with no changes to the total n u m b e r o f facilities and is increasing concavely to a score o f [00 when 100°~ o f the facilities were not changed during the year. In this case a = 2 . 5 4 , b = 4 . I8 and c = I.

4. P E R F O R M A N C E

MEASURES

The various weights and functions are incorporated into an evaluation model which pro-

424

Spie~ler et a[.--Ez'aluatin,~ Per!ormance Capahdit 3

duces measures for the performance level of activity i, the performance level in station k and the overall performance of the system. These measures are given for a specific set of x-values for the parameters which determine specific values t',j for the evaluation functions.

(I) Performance level of activity i Performance level results from:

P, = Y ,,.,,L.,,

is)

/

where P~ is the performance level of activity i over the entire system. It does not take into consideration the relative participation of individual stations in this activity. (2) Aggregated performance level relative to station k Aggregated performance level is defined as:

ever. t, = t~z= 3 and t~ = 12 indicates that activity 13 is far more important than 2 and 14 which may motivate station 5 to consciously devote more effort to activity 13 despite its relative low involvement. In other words, as far as the overall system is concerned management would rather have station 5 work harder on activity 13. On the other hand, one may argue that station k concerns (or should concern) itself mostly with activities in which it plays a larger role: and for this purpose equation (9) is more appropriate. Whatever the company's policy is, top management may wish to clarity it to the organizational units by disclosing the formula for &.. (3) Overall perJbrmance of the entire system Overall performance is given by:

T=F'ty, where Sk represents the aggregated level of system performance across all of the activities in which station k participates, weighted by the relative involvement of station k in each activity. This score cannot be regarded as an unequivocal measure for the performance level of station k, since a low or high level of perforrnance in each one of these activities may be attributed to other stations which take part in this activity. However, since S,. is a weighted aggregate relative only to the activities in which station k participates, a low S,. score may be taken as an indication that further checking on the performance of station k is needed. The division by a~ normalizes the a,k weights across the activities. Another possible measure for the aggregated performance of station k may take into consideration the relative importance of the activities in which station k participates: S~ = ~ (ae,/~a,k) (t,/ '

- t ) P,.

(10)

i:g',k ~ 0

The justification for that measure is based on the argument that station k tries harder on those activities which are known to be more important. For instance, station 5 is involved only in activities 2, 13 and 14 with a2.5 = at3.s = 10, a~<5 = 20. Hence, by equation (9) half of that station's performance would be evaluated from the performance level of activity 14 (PLy). How-

(II)

where T is an overall performance measure for the entire system obtained as a weighted average of the performance level of the activities weighted by their relative importance for the overall performance of the system. Since T is an overall numerical measure, it can be used for correlation analysis with more objective performance measures which are based on outcomes, such as volume of work actually performed. Such an analysis provides a tool for evaluating the validity of our model as a predictor for the system's actual performance. Implicit in the above additive measures is the assumption of lack of interaction effects between the parameters (equation 8) and between the activities (equations 9, 10 and 11). Such an assumption rarely holds in the theoretical sense. In order for our measures to serve as a reasonable approximation, we tried to minimize the violation of this assumption by choosing and defining parameters and activities as independent as possible. For alternative scoring measures and further discussion of the additively assumption see [4] and [3]. The model may also handle binary parameters as in the case of a station whose involvement is strictly limited to that of authorization or non-authorization. In such a case the x-values are 0 and 1 and the V,i(x) function is not continuous. The implications, however, are only technical and not conceptual.

Ome'ca. Vol. 11, Y;o. 5

5. ASSESSING M O D E L P A R A M E T E R S Johnson & Huber [3] review, categorize and compare techniques for assessing the parameters of a multi-attribute utility model. Using their classification, the technique used in our study belongs to the 'direct methods' category. This decision was primarily motivated by the large number of parameters that had to be assessed. Initially an alternative method was tried based on Saaty's model [6]. While responding favorably to Saaty's approach, our group of decision makers were bothered by its slowness as compared to the direct approach which was therefore adopted in this study. Model components were assessed in a three phase procedure. In the first phase, each of two decision makers that belong to the same department was given a description of the model and its parameters and a set of forms similar to Tables 1 and 2. They were requested to indicate by check marks (a) whether a given station participates in a given activity (Table 1) and (b) whether a given parameter contributes to the performance of a given activity (Table 2). In phase two, decision makers were asked to fill in specific weights for a,, and % to replace the check marks. Upon returning these forms, a session was held to discuss parameters for which wide differences exist. Some of the differences stemmed from a different understanding of an activity or a parameter. This session helped to resolve these differences and to achieve a better definition of the parameters. Final values representing the department view were reached, either by taking an average value or by resolving the conflicting views. (See, however, Note (1) below.) In the third phase the evaluation functions Vo(x) were estimated by first identifying the characteristics of the function, such as concave, S-shape or linear. Then the range of each function was determined, that is, the x-values for which V,j(x)= 100 and 0. Additional x-values within the range of each function were identified together with the score they should yield for Vii(x). At this stage the mathematical function which best fitted the general shape and specific selected x-values was chosen. Several tuning sessions were necessary to insure that the mathematical functions reflect the view of decision makers regarding the evaluation functions.

425

Notes Several final notes regarding the assessment techniques are in order: (1)

In our model we are not necessarily seeking a consensus. On the contrary, we wish to identify those areas where major differences exist, since this may highlight the reasons for different opinions on the system's performance. For instance, a significant disagreement on the relative involvement of stations in a particular activity may explain the differences a m o n g decision makers regarding the reasons for the low performance on this activity. However, we must make sure that these differences stem from a different perception of the organization and its activities and not from misunderstanding of the parameters" definition.

(2)

The aik values represent a combination of the relative length of time and manhours that station k invests during the entire process of performing activity i. For activities that are fairly modular, e.g. sequential, a,k may be relatively easily judged. A more elaborate assessment method may be required for activities that are carried out by several stations simultaneously. Such a method may be based on further decomposition of the activities and measuring the stations responsibility at a lower level where responsibility boundaries may be more easily identified.

(3)

In assessing the Vii(x) evaluation functions, decision makers were requested to bear in mind the same 0 to 100 scale across all i and j. Consequently, the minimum value for several functions did not reach the bottom level of zero, e,g. the minimum value for P6.t~ is 50. This implies that if two parameters have the same weight for a given activity, e.g. w6.a = w
Spiegler

426

et

al.--Evaluating Performance Capability

value of 0 than P~ v, ill increase by 0.08"(100-50) = 4, while changing "date of equipment orders" ( j = 4 ) from "worst' to 'best" scores (24 to - 2 4 ) will increase P, by 0.08"(100 - 0) = 8. When an evaluation function did not reach the 0 or 100 score for feasible values of the parameters, decision makers were made aware of that significance, and in most cases they indicated that this was precisely what they wished.

ment successive runs by varying certain parameter values and observing the effect on performance.

Output For each set of x-values three basic outputs: Pi, S~ and T are always produced. A more detailed output which includes any, or all, of model components such as w,/, az~.or V0 values for any i and j is optional at the user's request. 7. T H E M O D E L AS DECISION AID

Spread factor

6. M O D E L I M P L E M E N T A T I O N Input

Computer implementation of the model requires two types of input: relatively fixed and variable inputs. The relatively fixed inputs are quantities fed into the model, usually once, for any given organizational structure. It includes w~,a~ and ti weights and the Vii(x) functions. These values are elicited in the manner described above. The variable input consists of x-values for each of the parameters defined in the system. The x-values are substituted in the Vo(x) functions to generate the level of desirability of parameter j in the performance of activity i. Variable inputs to the model are obtained from decision makers via a questionnaire designed for that purpose (Appendix). This questionnaire provides a brief description of each parameter, range of the measuring scale and the definition of x (absolute quantity, ratio, percentage). Filling in the questionnaire does not require familiarity with the model. The computer program provides also an interactive mode for input of the x-values for those decision makers who are familiar with the model and desire to experi-

As a tool for planning and performance evaluation, the model enables decision makers to test various aspects and constraints of the system before their actual introduction to work. One such aspect is the introduction of a 'spread factor' into the system. This concept is briefly discussed and analyzed below. o/ Working with a spread factor of size k (/o) means that IWRDS orients itself towards planning and implementing projects within a budget ofk~o above the approved budget. Those favoring the spread policy argue that in order to perform at the approved budget level, the sys.o/ tern must act as if the budget is k/o higher. The reason is the possibility that some of the projects included in the approved budget may be delayed or cancelled because of objective circumstances (e.g. bad weather). In such cirumstances the allocated budget would not be fully utilized. Those who oppose it argue that spreading efforts geared for a budget of size x + k ~ may cause damage and waste of resources for projects which are started but not completed and thus lead to inefficient use of system capabilities. These opinion differences, stemming from an intuitive understanding of the policy,

TABLE 5. SPREAD FACTOR (k = SPREAD FACTOR ( o ) , X* = X AFTER SPREAD FACTOR APPLIED j

Parameter

Changes

Reason

I

Budget size

x * = x ( l + k/100)

Budget is increased

2.3,4

Time parameters

x * = x - k,'10

Each 10'~, spread is equivalent to one m o n t h earlier scheduling

13,14

N u m b e r o f buildings, n u m b e r o f contracts

,'¢* = x ( I + k q 0 0 )

More facilities/contracts pertbrm ~

17

Inventory to equipment ratio

x* =

22.23,24

W o r k changes

x* = x ( I +

x/(l + k,'100) k./ 100)

Decrease in existing inventory relative to the increased budget Less actual changes to be m a d e relative to budget

I Note that an increase in some parameters reduces the performance of the system, i.e., more buildings on the decreasing curve in Fig. le yields a lower score.

Omega, Vol. 11. 3,'o. 5

indicate that the true meaning and consequences of the policy, in our case spread policy, requires some additional clarification. Recognizing that a spread factor k implies changes in the parameter values which determine system performance, our model may be used to test the merit of a spread policy and its effects on performance. Table 5 summarizes the effect of a spread factor on the relevant parameters. For instance, with a budget o f x + k°;, the number of facilities (parameter 13) is increased proportionally, while tbe ratio of inventory to budget (parameter 17) decreases. Section 8 summarizes the experiments performed with spread policy. The somewhat surprising results brought forth the real significance of the policy, in terms of performance, and helped clarify the opinion differences. It/Tzat-~ situations The analysis of what-if situations is an important feature of the model as a decision aid. The following list is a sample of what-if questions, illustrating changes in each parameter group: budget, time, manpower, complexity, equipment, managerial abilities and plan changes: (I) Approved budget is increased from 60 to 850o of the requested budget. (2) All activities are advanced six months. (3) Overall planning of projects is completed six months earlier. (4) Formal approvals (administrative) given three months earlier.

are

(5) Planning manpower is increased from 60Yo of desired number to 100°/o. (6) Complexity of development program is reduced as a result of a decrease in number of facilities from 850 to 500.

427

(10)Number of facilities without changes is increased to 80°,,. Each of these changes attempts to isolate the effect of a change in a particular parameter or set of parameters on the performance of the system and its activities. Such changes need not be tested one at a time. Any combination of several concurrent changes may be introduced for evaluation by the model. For example, a combined run of questions 2, 5, 8 and 10 was attempted. That is, the system was asked to give the performance score if all activities were completed six months earlier, manpower increased, inventory increased and number of design changes decreased.

Conflict resolution When major differences exist between two decision makers, be it with regard to the overall performance or the performance of individual activities or units, we may use the model to pinpoint these differences by asking each decision maker to specify his own weights and values for the parameters. Comparing the two sets we eliminate those elements on which a consensus exists and focus on the differences. Such an analysis channels the discussion to specific issues. Disagreement about parameter values may be resolved by going to the data room; e.g., when was the equipment ordered. Disagreement about the weights given to parameters may lead to very useful discussions on the structure of the organization, the role of each unit and the scheduling of activities. Indeed, whenever such discussions arose in our project they revealed different perceptions regarding 'who is responsible for what' and what is the impact of a given parameter on a given activity.

8. E X P E R I M E N T S

(8) Inventory is increased to 50~o of budget.

Three sets of data were used to calibrate the model and to check its basic validity. These data which were obtained from two decision makers provide x-values describing the following situations:

(9) Managerial ability of PMU is increased to 80.

Set 1: operation under 'good' achievable conditions.

(7) Inventory is decreased from 20 to 10~o of budget.

Spiegler et al.--Evaluating PetT~)rrnanceCapability

428

Set

2: operation under ideal-utopic conditions.

optimal

through

Set 3: current (normal) operation of the system.

Table 6 contains the specific x-values used as input for each set and the input for the what-if simulation runs. Each set ran twice, once with the actual x values and then with a spread factor of k = 20%. Table 7 summarizes the performance on each activity and the overall performance as obtained from the three sets of data. These results are in line with those expected by the decision makers who generated the x-values. From the scores presented in Table 7, several interesting conclusions may be drawn with regard to spread policy. As intuitively expected, overall performance of the system increases with the introduction of a spread factor. For those favoring the spread policy it was somewhat

surprising, however, that the increase is not higher than shown: a score of 85 compared to 81 for Set 1 and 59 compared to 56 for Set 3. (Under optimal conditions (Set 2) no increase in overall performance was obtained since, under such conditions, ff,j(x) functions are very insensitive.) The low sensitivity of the model to spread policy may be explained by the fact that not all parameters contribute positively to the performance level with the change introduced by the spread factor. For example, the spread factor improves performance by increasing the budget or advancing dates; however it reduces performance if more facilities are to be constructed (parameter 13), as V,j for that parameter decreases with higher x-values (see Fig. le). Examining Table 7, we observe a strong increase in the score for activities 3 and 4 (preparing budget and ratifying development

TABLE 6. INPUT TO EXPERIMENT RUNS (x-VALUESI I

Sets Parameter

(I)

(2)

(3)

(1)

3,4 5,6.7,8,9,10,13

85 85

85 70

60 50

85

0 0 9

-3 -6 - I

6 0 10

0 -6 4

-4 -6 0 0 -6 0 l 4 -6

-12 -9 -6 -3 -6 0 0 4 -6

-3 2 4 5 6 6 9 15 6

-9 -4 -2 -1 0 0 3 9 0

(j)

5 6 9

4 5 6

What-if runs

Relevant activity (i)

I 2 3 4 7 8 10 11 6,10

3

0

6

7 8 9 10 11

100 90 100 100 100

100 100 100 100 100

80 150 60 80 60

12 13 14 15

25 500 1000 6

0 500 700 4

20 850 2000 3

16 17 18

35 50 25

30 15 0

35 20 10

19 20 21

80 80 40

I00 100 15

70 70 10

22 23 24

80 80 80

100 100 100

60 20 90

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9) (10)

(tl)

0 -6 4 -9 -4

-9 -4 -2 -1 0 0 3 9 0

-1

I00

100

500

50

Baseline run for what-if is Set 3. Inputs for what-if runs are shown only for different x-values from the baseline. * Parameter j = 5 takes 12 values for spread over year. N o t shown.

50 80

80

80

211 45

92 100 49 53

9~ 94

88

84

48 36 32 30 28 90

85 65 (~0 66 67 99

72 57 59 60 62 94

Advatlc¢ ('orllla[ approval 3 i)lt)llths hnprove manpower Io 1003, I)ecreasc facilities Io 500 Decrease i n v e n t m y to I()'~;i Increase inventory to 5()i~, Imln-Ove nlanagemenl ( P M t l ) to 80 Increase no design change It) 80~!,,

('olllbille rtHis 2,5,8,11t

Spread Ii~ctor I()'. .';plead I";ICIoF 20".

Spread factor 30','i, Spread factor 40'.I.,

Optimal operaliun

II

12

14 15

3') 39 48 48 39 39 54 39 39 3q 39

63 63 66 66 63 70 65 63 63 6t 75

56 57 63 58 59 58 59 55 58 57 58

4 5 6 7 8 9 I(I

(2)

(1)

Advance plans 6 inOlllJls

13

57 61

89 88

88

84

(6)

52 51

92 93

86

84

(7)

63 67

98 99

87

85

(8)

38 56

95 95

69

69

(9)

48 48

84 83

86

89

(I0)

93

30 42 56 70

44

19 19 19 19

19

19 34 44 19 44 19

(3)

92

32 45 59 72

45

20 35 45 45 45 20 20 20 20 20 2(I

(4)

96

51 53 55 57

72

49 50 59 53 53 55 53 49 49 50 57

(5)

YAIl~l.li 8, RI:SULIS OF ~,VItAI-II: FXPIiRIMliNIS

19 42

~)3 100

74 96

(5)

system

Overall

39 32

9(1 92

94

(4)

3

63 66

99 98

67

79

(3)

73

(2)

Baseline (Sel 3) Increase m budget 1o 85';., Advance activity 6 months

What if

,~'t't 31 nOllllLll (currenl) Aclual .v Splead l'aclol k 211

Spread filclOl k : 20

Set 2: ~ptimal (tllopic) actUill ~.

81 89

(l)

0 I 2

Run

[ )i| Ill ScIs

Set 1: g~md (achievable) Actual Spread factor k = 20

Activities

89

59 61 62 61

75

57 57 65 57 57 61 60 55 05 58 57

(6)

57 b(I

93 92

88

9(I

(II)

]'AB[.I: 7. ()~.IR&l l S'ISIIM ANt) AUIIVIIIIS I,IRII)RMAN( I

74 72

94 93

82

84

(13)

92

52 51 50 51

,'*8

52 52 50 52 52 56 56 52 52 51 (,0

98

66 07 70 74

80

63 63 80 63 63 6~ 63 (,3 611 69 6t

Aclivitlcs (7) (8)

77 76

95 94

85

87

(12)

75 72

99 98

79

80

(14)

95

57 % 54 54

64

58 58 61 58 58 58 63 58 ~,8 59 61

(9)

84

,17 ,18 ":,0 ~,4

79

48 48 69 48 48 48 52 45 58 ,18 -IF,

(l(I)

h6 >9

94

94

SI 85

93

62 63

hO

58

57

57 57 63 57 57 57 63 57 57 ~,7 37

(11)

Overall Systenl P¢ll()rl)lallCe T

95

77 7t, 75 75

76

77 77 74 77 77 77 79 77 79 78 77

(12)

94

73 /2 71 71

78

7-1 7,1 75 77 ]4 7(, 7', 7~*

70

74 74

(1~)

99

7 12 7I 69

7-1

75 75 74 75 7:, 75 80 7:~ 75 7", 1",

(14)

~Jj

430

Spiegler et al.--Eraluating PerJbrmanceCapability

plans) as a result of spread policy. This is true activities (j = 2,3) by six months (run 2). Again, for each of the three sets of input data. One advancing schedules may be achieved by interreason is that the performance of these activities nal organization changes as opposed to a spread is not affected by parameters which affect per- policy which requires less easily obtainable exformance negatively (such as number of facili- ternal approval. ties) when spread factors are introduced. On the other hand, early scheduling of these activities (time parameters j = 2,3) resulting from the 9. S U M M A R Y spread factor contributes positively to the perA model for evaluating the performance formance. From a managerial point of view, this is an important finding as early scheduling may capability of a multi-organization system was The model is based on be achieved by internal means such as or- presented. ganizational changes or tighter control over an hierarchical-modular decomposition of the varearlier completion of plans, rather than ap- ious factors which determine system perplying a spread policy. Since a spread policy formance up to the level where they can be needs the approval of external authorities, any measured quantitatively, or at least qualsolution which may achieve the same results by itatively, in an objective or subjective manner. internal changes may be preferred (depending Performance measures are obtained by incorpoon its cost). rating these components into multi-attribute This point illustrates the model's ability to evaluation measures. The model was implemented in a computer help resolve opinion differences. By realizing, as a result of computer runs of the model, that a program and tested successfully with several sets spread policy is really equivalent in terms of of data. All inputs are easily changed, which performance to an advance of the planning makes it possible to run the model with a wide activities, decision makers now understand the range of data obtained from decision makers significance of such policy. The discussion, pre- who agree about the structure of the system, but viously founded on intuitive factors, is now disagree about parameter values or the dechanneled into a fact-oriented framework in pendence of activities on various parameters. which proposals for action are more vivid and As a decision aid the model may be used for: objective. Table 8 shows the results of the sample (1) identifying activities and organizational what-if questions mentioned above. Line 0 conunits in a system where an improvement in tains the baseline (Set 3) results under current the performance level is required; operation; while consecutive lines show the resuits after various changes have been intro- (2) obtaining quantitative performance scores duced. The results indicate that overall perfor system activities, stations and overall formance is fairly insensitive to changes in one system performance; parameter, though the performance in specific activities may change significantly (runs 1 to (3) evaluating the impact of potential changes 10). However, when several changes are introin operational strategy or structure and duced simultaneously, as in run 11, the overall comparing between system set-ups before performance also changes significantly; in our and after changes or between past and case from 56 to 72. ~ current system structures; We also ran four experiments to show the effect of a spread policy. Run 15, for example, (4) resolving conflicts by uncovering specific indicates that applying a spread policy with reasons for different opinions regarding the level of performance. k =40~o is basically identical in overall performance to advancing the schedule of planning By using the model, management is able to direct discussions concerning the performance I lt can easily be proven that the activity performance of the system, its structure and work plan into difference from the baseline following a combined change goal-oriented channels focusing on specific variequals the sum of activity performance differences from each of the separate changes. ables and parameters of performance.

Omega, ~'ol. I1, Vo. 5 REFERENCES 1. EDWARDS W (1977) How to use multiattribute utility measurement for social decision making. IEEE Trans. Syst. Man Cvbernet. 7, 326-339. 2. HUBER GP (1974) Multi-attribute utility models: a review of field-like studies. Mgrnt Sci. 20, 13931402. 3. JOHNSON EM & HUBER GP 11977) The technology of utility assessment. IEEE Trans, Syst. Man Cvbernet. 7, 3 l 1-325.

43[

4. KEE.x'Y RL & ILaiFEa H {1976) Decisions with 31ultiple Objectives: Pre]~rences and Value Tradeoff;. John Wile?, New York. 5. MURDtCK RG ( [97l ) Mathematical 3,1odelin~, in .kIarketin¢. Chap. 7. lntext Education Publishers. 6. Saap( TL I1977) A scaling method for priorities m hierarchical structures. Jl math. P~vchol. 15, 234-281. FOR CORRESPONDENCE: Dr Israel Spiegler. School oj Management. Boston L'nit'ersitv. 704 Conzm,mu'ealth Avenue. Boston. MA 02215. USA.

ADDRESS

APPENDIX P E R F O R M A N C E EVALUATION MODEL Questionnaire Jbr determining parameter values (x ) Assign an x-value for each parameter as described. Note the range of valid values for each parameter. ,4ctirities Parameter Explanation 3,4 l Ratio of approved to requested budget (30. between 50, 100) 5,6.7,8,9. t0.13 Ratio of planning budget to requested budget ("o. between 50, 100) End date of 80':~, of': (month, between - 2 4 . 24) 5 Detailed planning 6 Purchasing 9 Contract preparation End date of: (month, between - 2 4 . 24) 1 Approval of overall plans 2 Preparing development plans 3 Preparing budget 4 Ratifying development plans 7 Electricity orders 8 Real-estate preparations 10 Equipment supplied II Electricity supplied 6,10 Date of equipment orders (month, between - 2 4 , 24) All relevant Give work spread over the year 12 values (in percentages) for the ratio performance in month i x , = overall performance ; y-''v~= 100 fiscal month 4 5 6 7 8 9 I0 1[ 12 1 2 3 x. All relevant Schedule overrun (months, between 0, 12) All relevant Manpower of SHM (subsidiary) existing x (3o between 50, 200) desirable All relevant 8 Manpower of BP (subsidiary) same as 7 (grade between 0, 100) All relevant 9 Planning manpower (grade between 0, 100) All relevant tO NTP, managing unit 9f.development manpower extstmg x (o{; between 16, 200) desirable All relevant II Supervision manpower (grade, between 0, I00) All relevant 12 Overall complexity percentage of non-standard facilities (between 50, I00) All relevant 13 Number of facilities in development plans (between 50, 1500) 14 All relevant Number of contracts in development plans (between 100, 5000) 15 All relevant Geographical location of majority of work (choose one) (1) Desert (2) Mt Negev (3) Northern Negev (4) Jordan Valley (5) West Bank, Gaza (6) Golan (7) Upper Galilee (8) North shore (9) Central shore (10) Interior plain. All relevant 16 Equipment to budget ratio (o~ between 0, I00) 17 All relevant Current inventory to equipment ratio ("~; between 0, 100) All relevant 18 Special equipment ratio to all equipment ("~; between 0, I00)

o'~ II 5

X

-U(./lLleS

m_

m

m

m

Spiegler et al.--Ecaluatz'ng Performance Capability

432 19

All relevant

20 21

All relevant All relevant All relevant

23

All relevant

24

All relevant

Managerial ability--NTP (managing unit of development) (grade, between 0, 100) Managerial ability of subsidiaries (grade. between 0, 100) Contractors to subsidiaries ratio (o~, between 0. 100) Changes in work plan number of facilities without changes relative to budget (° b between 0, I00) Design changes number of facilities without changes in design relative to budget (o~ between 0. t00) Design data changes number of facilities without changes in design data relative to budget (% between 0, 100)