Performance measurement system design: Should process based approaches be adopted?

international journal of

product[on ELSEVIER

economics Int. J. Production Economics 46 47 (1996} 423 431

Performance measurement system design: Should process based approaches be adopted? Andy Neely*, John Mills, Ken Platts, Mike Gregory, Huw Richards Mam~Jitcturing Engineering Group, Universi(v ()f Cambridge, Cambridge, UK

Abstract

Performance measurement system design is a topic of increasing concern to both academics and practitioners but is complicated by its multi-dimensional nature. When designing measurement systems managers need to resolve issues such as: conflicts between performance measures; the appropriate balance of internal and external measures; the linking of measures and strategy; etc. One way of overcoming the inherent complexity of performance measurement system design might be to employ structured design methodologies. This paper reports the preliminary results of a study into the use of structured processes for the design of performance measurement systems in the UK. Over 850 companies from a variety of industries were surveyed and data on the formality of their performance measurement system design processes were collected. The data show that although few firms use structured methodologies for performance measurement system design, those that do often find it significantly easier to: (a) decide what they should be measuring; (b) decide how they are going to measure it; (c) collect the appropriate data and (d) eliminate conflicts in their measurement system. Kevwords: Performance measures; Performance measurements; Process approach; Balanced scorecard; Manufacturing

strategy

1. Introduction

The fact that performance measures influence behaviour and can therefore be used to induce certain courses of action has long been recognised [1-5]. This, coupled with an increased awareness that the traditional measures of manufacturing performance often encourage inappropriate behaviour, has stimulated m u c h interest in the subject This paper was produced during the research project, Manufacturing Strategy and Performance Measurement, which was sponsored by the CDP Directorate of EPSRC under grant number GR/H21470 and GR/K53086. * Corresponding author.

[6, 7]. Various authors, for example, have suggested that performance measures can be used to affect the implementation of strategies [8 10]. Others have argued that performance measures are an integral part of the strategic control loop, without which managers cannot tell whether they will achieve their objectives [11]. In addition the nature of the market place is changing. D a t a gathered during the manufacturing futures survey support the notion that firms are seeking increasingly to build competitive advantage t h r o u g h the nonfinancial dimensions of manufacturing performance [12, 13]. Given this, and accepting that m a n y of the traditional measures of manufacturing performance are obsolete, a key question for the

0925-5273/96/$15.00 Copyright ~': 1996 Elsevier Science B.V. All rights reserved Pll S 0 9 2 5 - 5 27 3 ( 9 6 ) 0 0 0 8 0 - I

424

A. Neely et al./Int. J. Production Economics 46 47 (1996) 423 431

1990s and beyond, is how can performance measurement systems which are appropriate for modern manufacturing firms be developed? One possibility might be to use a process based approach to performance measurement system design similar in style to the manufacturing audit described by Platts and Gregory [14]. This paper explores this notion more fully and seeks, through a literature review and survey, to: (a) Identify what a process-based approach to performance measurement system design might constitute. (b) Establish whether U K small and medium sized manufacturing enterprises (SMEs) employ such processes. (c) Determine what the perceived benefits of these processes are. The remainder of the paper has been split into four main sections. The first reviews the relevant literature thereby identifying what a process based approach to performance measurement system design might constitute and what one could contribute. The next two sections use the data gathered during a survey of over 850 U K SMEs (small and medium sized manufacturing enterprises) to address the question of whether process based approaches to performance measurement system design are currently being used and if so what benefits they provide. In the final section some of the questions that this research raises are examined.

2. B a c k g r o u n d l i t e r a t u r e

Performance measurement is a topic which is often discussed but rarely defined. Literally it is the process of quantifying action, where measurement is the process of quantification and action correlates with performance. According to the marketing perspective organisations achieve their goals, that is they perform, by satisfying their customers with greater efficiency and effectiveness than their competitors [15]. The terms efficiency and effectiveness are used precisely in this context. Effectiveness refers to the extent to which customer requirements are met, while efficiency is a measure of how economically the firm's resources are utilised when providing a given level of customer satisfaction.

This is an important point because it not only identifies two fundamental dimensions of performance, but also highlights the fact that there can be internal as well as external reasons for pursuing specific courses of action [16]. Take, for example, one of the quality related dimensions of performance, product reliability. In terms of effectiveness, achieving a higher level of product reliability might lead to greater customer satisfaction. In terms of efficiency, it might reduce the costs incurred by the business through decreased field failure and warranty claims. Hence, in a business context, performance can be defined as the efficiency a n d e f f e c t i v e n e s s o f action. Given this definition three others follow: - P e r f o r m a n c e m e a s u r e m e n t : the process of quantifying the efficiency and effectiveness of action, - P e r f o r m a n c e measure: a metric used to quantify the efficiency and/or effectiveness of action, - P e r f o r m a n c e m e a s u r e m e n t s y s t e m : the set of metrics used to quantify the efficiency and effectiveness of actions. Fig. 1 shows how a set of performance measures, a performance measurement system, can be examined at three different levels: ( a ) t h e individual performance measures; (b) the performance measurement system as an entity; and (c) the relationship between the performance measurement system and the environment within which it operates. Table 1 uses this framework to categorise the issues that have to be considered when designing a performance measurement system according to [17-27, 10, 5].

E "

Fig. 1. A framework for performance measurement system design.

A. Neely et al./Int. J. Production Economics 46 47 (1996) 423-431

425

Table 1 Issues to consider when designing a performance measurement system

Individual measures

Measures should be clearly defined/easy to understand

(Fortuin, 1988: Hronec, 1993).

Measures should be purposeful Measures should be practical. They should have the appropriate scale

(Maskell, 1989). (Crawford and Cox, 1990~.

Measures should form part of the control loop

{Bungay and Goold, 1991~

Measures should be failsafe/selfchecking Measures should be cost effective

(Fortuin, 1988; Hayes et al., 1988).

Performance measures should be integrated over both the functions and hierarchy The system should provide data for monitoring past and planning future performance Performance measurement system

Performance measurement system & the environment

(CAM-l. 19911. (ICAS, 19931.

The measurement system should provide a balanced picture of the business

(Kaplan and Norton, 1992).

The measurement system should show how results are a function of determinates

(Fitzgerald et al., 1991).

The measurement system should not contain any measures which conflict with one another

(Fry and Cox, 1989).

The performance measurement system should reinforce the firm's strategies The performance measurement system should match the firm's culture The performance measurement system should match the existing reward systems The performance measurement system should provide data for external comparison

(Skinner, 1971~ Maskell, 19891.

T a b l e 1 highlights the complexity of the perform a n c e m e a s u r e m e n t system design task, which n o t only involves the selection a n d definition of an a p p r o p r i a t e a n d practical set of measures, b u t also their i n t e g r a t i o n b o t h with one a n o t h e r a n d the wider e n v i r o n m e n t , c o n s t i t u t i n g the rest of the org a n i s a t i o n a n d indeed the m a r k e t place itself. The p r i m a r y benefit to be gained by e m p l o y i n g a structured a p p r o a c h to performance m e a s u r e m e n t system design then should be that such a n a p p r o a c h will provide a m e c h a n i s m for m a n a g i n g the complexity of the performance m e a s u r e m e n t system design process. M o r e specifically, it can be hypothesised that o r g a n i s a t i o n s with structured approaches to p e r f o r m a n c e m e a s u r e m e n t system

(IPM, 1992).

(Hayes et al., 19881.

design will find it easier t h a n those which do not, to: (a) Decide what they should be measuring. (b) Decide how they are going to measure it. (c) Collect the a p p r o p r i a t e data. (d) E l i m i n a t e conflict in their m e a s u r e m e n t system.

3. Research methodology T o test these hypotheses 858 small to m e d i u m sized m a n u f a c t u r i n g enterprises (SMEs) in the U K were surveyed in late 1992. The definition of an S M E was any firm with less t h a n 400 employees a n d such firms were targeted because the ultimate

426

A. Neely et al./lnt. J. Production Economics 46-47 H996) 423-431

objective of the authors' current research is to produce a "workbook" to help managers in SMEs develop appropriate performance measurement systems and manufacturing strategies. The questionnaire was designed in accordance with Dillman's Total Design Method [28]. It consisted of three main sections. In the first the respondent was asked for information on: how the firm competed; what it measured; the formality of its manufacturing strategy development process; the formality of its performance measurement system design process; what problems it encountered when developing manufacturing strategies; and what problems it encountered when designing performance measurement systems. The second section contained free form questions which asked the respondent for more detailed information on the processes they used to design their performance measurement systems and develop their manufacturing strategies. Finally, in the third section, the respondent was asked to state the company's turnover, its size, the industrial sector in which it competes, and the type of manufacturing organisation used. Once the survey instrument had been developed it was pretested with three groups of people: - academics who specialised in questionnaire design; academics who might use the data once it had been collected; - senior industrialists who were likely respondents. This pretesting consisted of a series of formal interviews designed to check whether the questionnaire was coherent. Following them the questionnaire was piloted with fifty companies randomly selected from data provided by Dun and Bradstreet. A number of those firms which did not respond to the pilot survey were telephoned and asked why they had not completed the questionnaire. In November 1992 the full survey was released. 858 firms were mailed. Each questionnaire was accompanied by a covering letter and sent to a named director. All the firms surveyed were believed 2 to meet the following criteria: (a) they

were manufacturing firms; (b) they employed between 150 and 400 people; (c) they had a turnover of greater than 7.5 million pounds; (d) they were based in the UK; and (e) they were associated with either the primary metal industries (SIC 33); fabricated metal products industry, except machinery and transportation equipment (SIC 34); industrial, commercial and computer equipment industry (SIC 35); electronic, other electrical equipment and components industry, except computer equipment (SIC 36); transportation equipment industry (SIC 37). These selection criteria were carefully chosen to ensure that data would be obtained from a wide variety of manufacturing organisations and industries. In the following section the results relevant to this paper are documented. These are based on a 40.1% response rate. This was achieved after two follow-up mailings; a postcard and a letter.

4. Survey results Figure 2 shows the percentage of respondents from each industrial group. Figures 3 and 4 show the respondent profiles according to the number of people they employed and their turnover, respectively. From these data it can be seen that the sample consisted mainly of firms with turnovers between 10 and 20 million pounds, and employing between 100 and 300 people, giving an average turnover per employee of approximately £75000. The proportion of respondents from each industrial sector is consistent with the original sample. To gather the data required to test the hypotheses listed earlier the questions shown in Tables 2

J ~ [] ~

F~ . ~ t

[] E ~ c

~l~e~t

• 2 These criteria were the ones used by D u n and Bradstreet to isolate the target sample from their database. As will be seen, however, changes in c o m p a n y performance meant that the actual sample contained some firms outside the original categories.

Meaals

T1~spor~on

Equipment

Fig. 2. Profile of respondents by industrial sector.

427

A. Neely et al./Int. J. Production Economics 46- 47 (1996) 423 431

50%' P

r

[ ] PrimaryMetals

C

30% ctc 40%20% n

Fabricatod Metals Im [] IndustrialEquipment

a

'[] ElectronicEquipment

g

[] TransportationF_,quipmcnt fit

0-100 101.200 201.3110 301400 401-500 >500N0~me Employees Fig. 3. Profile of respondents by industrial sector and number of employees.

P 5O% r c 30% C

[] PrimaryMemls

C

[]FabricamdMetals

n

t

[] ~

a

g

e 1o% 0%

r~ui~t

[ ] Trading.on Equipment

0-10

10-2020-3030..4040-50>50Nomspome Tumov~ in £ Million Fig. 4. Profile of respondents by industrial sector and turnover.

and 3 were asked. The questions shown in Table 2 elicited data on how easy each respondent found it to: (a)decide what they should be measuring; (b) decide how they could measure it; (c) collect appropriate data; and (d) eliminate conflicts in their measurement system. By averaging the data gathered in response to the ten questions shown in Table 3 a proxy measure of the formality of each respondent's performance measurement system design process was produced. It was decided to use this proxy measure rather than simply ask: "do you think you have a structured approach to performance measurement system design?", as it was thought few of the respondents were likely to

the

have a benchmark against which they could assess process formality. Figure 5 shows the distribution of process formalities across the industry sectors surveyed. Once these data had been generated they were used to split the respondents into two groups. Those that average 4 or more in response to the ten questions shown in Table 3 (32% of the sample) were assumed to have a formal performance measurement system design process, while the remainder (68% of sample) were said to have informal processes. The responses of each sub-group to the questions listed in Table 2 were then averaged by industry sector and a one tailed t-test used to compare the

the

428

A. Neely et al. ~Int. u( Production Economics 4 6 - 4 7 (1996) 423-431

Table 2 Questions on the case of designing performance measurement system Q-4 To what extent do you agree or disagree with each of the following statements as applied to your company? (Please circle the appropriate number) Strongly Disagree

Disagree

Neither agree nor disagree

Agree

Strongly Agree

1. We find it easy to decide which of the financial aspects of manufacturing we should be measuring.

1

2

3

4

5

2. Having decided which of the financial aspects of manufacturing we should be measuring we find it easy to design the performance measures for doing so.

1

2

3

4

5

3. We find it easy to collect the data we need to measure the financial performance of manufacturing.

1

2

3

4

5

4. We find it easy to decide which of the non-financial aspects of manufacturing (quality, lead times, etc.) we should be measuring.

1

2

3

4

5

5. Having decided which of the non-financial aspects of manufacturing we should be measuring we find it easy to design the performance measures for doing so.

1

2

3

4

5

6. We find it easy to collect the data we need to measure the non-financial performance of manufacturing.

1

2

3

4

5

7. We find it easy to make sure that our financial and non-financial measures of manufacturing do not conflict.

1

2

3

4

5

8. We find it easy to make sure that the measures we use in manufacturing do not conflict with the requirements of our customers.

1

2

3

4

5

9. We find it easy to make sure that the measures we use at the different levels of our manufacturing function do not conflict.

1

2

3

4

5

1

2

3

4

5

10. We find it easy to make sure that the measures used in manufacturing do not conflicit with those used in other functions.

sample means shown in Table 4. The resultant data are shown in Table 5. As Table 5 shows for all the industrial groups, with the exception of the primary metals category, all the hypotheses proved true with at least 90%, and often considerably higher, confidence levels. That is the data supports the assertion that firms which utilise formal process based approaches to performance measurement system design find it easier than those that do not, to: decide what they

should be measuring; decide how they are going to measure it; collect the appropriate data; and eliminate conflict in their measurement system.

5. Discussion This research raises a number of questions. First why do formal processes not appear to simplify the performance measurement system design task for

429

A. Neely et al./'Int. J. Production Economics 46 47 (1996) 423-43l

Table 3 Questions on the formality of the performance measurement system design process Q-5 I o what extent do you carry out the following in your company? (Please circle the appropriate number) Not at all

Just a little

Moderate amount

Quite at lo1

A great deal

I. We usually introduce new measures when we encounter a particular problem.

1

2

3

4

5

2. We check that new performance measures do not conflict with any existing measures.

1

2

3

4

5

3. We regularly review whether what we are measuring is what we should be measuring.

1

2

3

4

5

4, When a measure becomes obsolete we usually delete it.

1

2

3

4

5

5, We formally document our performance measures.

1

2

3

4

5

6, We clearly define who is responsible for acting on each performance measure.

1

2

3

4

5

7. We check that our measures encourage people to do what we really want them to do.

I

2

3

4

8. We review our measures at the same time as we update our manufacturing objectives.

1

2

3

4

5

9. We quantify our financial objectives for manufacturing.

1

2

3

4

5

1

2

3

4

5

10. We quantify our non-financial objectives for manufacturing. Note: In lhe proxy measure of process formality item 1 is negatively scaled.

5O []

Me s

FabricatedMetals IndustrialEquipment i Z

ElectronicEquipment 10

TransportationEquipment

0 0-0.95

I-1.95

2-2.95

3-3.95

4-4.95

Proxymeasureof processformality Fig. 5. Distribution of process formality according to the proxy measure.

f i r m s t h a t o p e r a t e in t h e p r i m a r y m e t a l s s e c t o r ? F r o m T a b l e 4 it c a n b e s e e n t h a t i n c o m p a r i s o n with the other industry sectors those firms in the primary metals sector with informal processes ap-

pear to find performance

measurement

system de-

s i g n e a s i e r o n a v e r a g e t h a n t h e c o m p a r a b l e f i r m s in the other industry sectors, while those with formal processes appear

t o f i n d it m o r e

difficult. Hence

430

A. Neely et al. ~Int. J. Production Economics 4 6 - 4 7 (1996) 423-431

Table 4 Mean response to Q4.x by industry sector and process formality Questions

Primary metals

Fabricated metals

Industrial equipment

Electronic equipment

Transportation equipment

Formal/informalprocess according to the proxy measure.

Informal process

Formal process

Informal process

Formal process

Informal process

Formal process

Informal process

Formal process

Informal process

Formal process

Decide what they should be measuring (Questions 4.1 and 4.4)

3.63

3.85

3.56

4.26

3.56

4.29

3.56

4.03

3.34

4.36

Decide how they are going to measure it (Questions 4.2 and 4.5)

3.38

3.55

3.10

3.94

3.21

3.81

3.18

3.65

3.08

3.57

Collect the appropriate data (Questions 4.3 and 4.6)

3.03

3.30

3.04

3.78

2.95

3.52

3.13

3.38

3.03

3.36

Eliminate conflict in their measurement system (Questions 4.7-4.10)

2.98

3.60

2.95

3.59

2.95

3.64

3.08

3.37

2.95

4.00

Table 5 Testing the hypotheses Hypotheses

Primary metals

Fabricated metals

Industrial equipment

Electronic equipment

Transportation equipment

Organisations with structured approaches to performance measurement system design will find it easier, than those which do not, to:

Sample: 32 with informal and 10 with formal processes

Sample: 52 with informal and 23 with formal processes

Sample: 73 with informal and 29 with formal processes

Sample: 60 with informal and 17 with formal processes

Sample: 32 with informal and 37 with formal processes

Decide what they should be measuring

t = 0.535 (No significant difference)

t = 0.660 (True-99.5% confidence)

t = 0.287 (True-99.5% confidence)

t = 3.067 (True-99.5% confidence)

t = 7.474 (True-99.5% confidence)

Decide how they are going to measure it

t = 0.675 (No significant difference)

t = 0.640 (True-99.5% confidence)

t = 0.541 (True-99.5% confidence)

t = 2.164 (True-99% confidence)

t = 1.433 (True-90% confidence)

Collect the appropriate data

t = 0.815 (No significant difference)

t = 0.748 (True-99.5% confidence)

t ~ 0.829 (True-99.5% confidence)

t = 1.587 (True-90% confidence)

t = 1.690 (True-95% confidence)

t = 0.903 (No significant difference)

t = 0.724 (True-99.5% confidence)

t = 0.548 (True-99.5% confidence)

t = 1.404 (True-90% confidence)

t = 7.408 (True-99.5% confidence)

Eliminate conflict in their measurement system

although the primary metals sector appears to have some property that makes performance measurement system design easier for it than the other industry sectors, it also has some unique problems that formal processes fail to simplify. The most

obvious explanation for these discrepancies is a function of the fact that the primary metals industry is primarily a process industry and hence is likely to rely on well developed measures, such as efficiency and machine utilisation. While this may

,4. Neely et al./lnt. J, Production Economics 46 47 (1996) 423 431

explain why primary metals companies appear to find it easier than the average to select and define appropriate performance measures, it does not explain why those companies in the primary metals industry that employ formal processes for performance measurement system design do not find the task {except for the elimination of conflicts) significantly easier than those which do not. This is a question that could be researched further. Leaving aside the primary metals industry, the second question that this research raises is that given that formal processes simplify the performance measurement system design task, then can the impact of such processes be quantified'? Take, for example, the fabricated metals sector. Table 4 shows that on average those firms that employ formal processes for performance measurement system design find the task easier than those that do not, by ~ 0.7 on a scale of strongly agree to strongly disagree. But what does 0.7 on this scale mean in terms of saved time and effort and also quality of the resultant measurement system? Further data would have to be collected before this question could be reliably answered. Suffice it to say, this survey has shown that firms which employ formal processes for performance measurement system design find it significantly easier than those lhat do not, to: (a) Decide what they should be measuring. (b) Decide how they are going to measure it. tc) Collect the appropriate data. td) Eliminate conflict in their measurement system.

References [13 Child, J. Organisation A Guide to Problems and Practice. Harper and Row, London, [2~] Hiromoto, T., 1988. Another hidden edge: Japanese management accounting. Har. Bus. Rev., 22 26. [31] Hopwood, A.G., 1974. Accounting and Human Behaviour. Prentice-Hall, Englewood Cliffs. [4] Hrebiniak, E.G. and Joyce, W.F., 1984. Implementing Strategy. Macmillan, New York. [51] Skinner, W., 1971. The anachronistic factory, Hat. Bus., Rev., 61 70. [6] Hall, R.W., 1983. Zero Inventories. Dow-Jones Irwin, Homewood, IL.

431

[7] Johnson, H.T. and Kaplan, R.S., 1987. Relevance Lost The Rise and Fall of Management Accounting, Harvard Business School Press, Boston, MA. [8] Dixon, J.R., Nanni, A.J. and Vollmann, T.E., 1990. The New Perti~rmance Challenge Measuring Operations for World-Class Competition. Dow Jones-Irwin, Homewood. IL. [9] Globerson, S.. 1985. Issues in developing a per[i)rmance criteria system for an organisation, Int. J. Prod. Res., 23: 639 646. [10] Maskell, B., 1989. Performance measures I\~r world class manufacturing, Mgmt. Accounting, 32 33. [11] Goold, M. and Quinn, J.J., 1990. The paradox of strategic controls, Mgmt. J., 11:43 57. [12] Miller, J.(i., Kim, J.S., DeMeyer, A., Fcrdows, K., Nakane, J. and Kurosu. S., 1991. Factories of the fllture, executive summary of the 1990 manufacturing futures survey, Boston ( niversity, School of Management. Manufacturing Roundtable. [13] DeMeyer, A., 1992. Creating the virtual factory. Report on the 1992 European Manufacturing Futures Survey. INSEAD December, 1992. [14] Platts. K.W. and Gregory, M.J.. 1990. Manufacturing audit in lhe process of strategy tk)rmulation, Int. J. Oper. Prod. Mgmt., 10(9), 5 26. [15] Kotler. P., 1984. Marketing Management Analysis, Planning and Control. Prentice-Hall, Englewood Cliffs, NJ. [16] Slack, N. 1991. The manufacturing advantage: Achieving competitive manufacturing operations, Mercury, London. [17] Bungay, S. and Goold. M.. 1991. Creating a strategic control system. Long Range Planning, 24~3): 32 39 [18] CAM-I Minutes of the Meeting held in Milan. Italy, 23 25 April 1991. [19] Crawford, K.M. and Cox, J.F., 1990. Designing performance measurement systems for just-in-time operations. Int. J. Prod. Res., 28(111:2025 ~2036. [20] Fitzgerald, L., Johnston, R., Brignall. S., Silvestro. R. and Voss, C, 1991. Performance Measurement in Service Business CIMA, London. [21] Fortuin, L., 1988. Performance indicators Why, Where and HovC~ Eur. J. Oper. Res., 34tl): 1 9. [22] Fry, T.D. and Cox, J.F.. 1989. Manufacturing performance: local versus global measures, Prod. Inv. Mgmt. J., 3012): 52 56. [23] Hayes, R.H., Wheelwright. S.C. and Clark, K.B.. 1988. Dynamic Manufacturing: Creating the Learning Organisation. Free Press, New York. [24] Hronec. S.M., 1993. Vital Signs Using Quality, Time, and Co,,t Performance Measurements to Chart Your Company's Future. Amacom, New York. [25] ICAS Measurement The Total Picture, The Institute of Chartered Accountants of Scotland, 1993. [26] IPM Performance Management in the UK An Analysis of the Issues, IPS/IMS, 1992. [27] Kaplan, R.S. and Norton. D.P., 1992. The Balanced Scorecard Measures that [)rive Performance. Har. Bus. Rev., 71 79.