- Email: [email protected]
Applying quality award criteria in R&D project assessment
Int. J. Production Economics 80 (2002) 119–128
Applying quality award criteria in R&D project assessment Ville Ojanena,*, Petteri Piippob, Markku Tuominena a
Department of Industrial Engineering and Management, Lappeenranta University of Technology, P.O. Box 20, FIN-53851 Lappeenranta, Finland b R&D Center, Valtra Inc., P.O. Box 557, 40101 Jyvaskyl a, . . Finland
Abstract Due to the problematic characteristics and complexity of R&D, the applicability of Total Quality Management (TQM) and quality award criteria in the assessment of a company’s R&D process is very challenging. This study concentrates especially on the application of the Malcolm Baldrige Quality Award (MBQA) Criteria to R&D assessments at project level. The applicability of quality award criteria in the assessment of R&D projects is first discussed with the help of a literature review at conceptual level. The meaning of different sub-areas of quality award criteria is analyzed from the point of view of R&D activities and single projects. The measures, performance criteria and concrete measurable aspects for R&D project evaluation are then derived on the basis of the analysis. In the empirical part of the study, the analysis of the utilization of criteria for R&D project assessment is discussed from the viewpoint of a manufacturing company that has successfully applied the Finnish National Quality Award Criteria based on the MBQA. The study gives examples of the derivation of new R&D project measures from the quality award criteria framework. r 2002 Elsevier Science B.V. All rights reserved. Keywords: Total quality management (TQM); Quality award criteria; Self-assessment; R&D projects; Performance evaluation
1. Introduction An important consequence of the introduction of systematic approaches of quality management, e.g. quality awards, is the increasingly widespread use of their models and criteria for company selfassessment. Several similar types of national criteria are used for assessment, e.g. the Malcolm Baldrige Quality Award Criteria, the European Foundation for Quality Management (EFQM)model, the Deming Prize and the Finnish National *Corresponding author. Tel.: +358-5621-2670; fax: +3585621-2667. E-mail address: ville.ojanen@lut.fi (V. Ojanen).
Quality Award Criteria. This study concentrates on application of Finnish criteria to R&D project assessments. The Finnish Criteria were formerly based on the MBQA, but from the year 2001 they are based on the EFQM-model. The framework utilized in this study is based on the MBQA. In a number of earlier studies, the Total Quality Management (TQM) philosophy has been argued to be an applicable approach also for the management of R&D. However, due to the problematic characteristics and complexity of R&D, the application of quality award criteria in the assessment of a company’s R&D process is very challenging (see e.g. Bellary and Murthy, 1999; Boyer, 1991; Fisher and Heywood, 1992; Kiella
0925-5273/02/$ - see front matter r 2002 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 5 - 5 2 7 3 ( 0 2 ) 0 0 2 4 7 - 5
120
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
and Golhar, 1997; Lovett, 1992; Patino, 1997). Companies have different types of R&D projects and in addition to the process level, suitable performance criteria are needed also at the project level. There is a need in many organizations to bring the criteria also to project-level assessment in order to effectively control and manage different types of single R&D projects and the whole project portfolio. The proposed systematic analysis approach of this study has several phases. First, the application of quality award criteria in the assessment and evaluation of R&D projects is discussed with the help of a literature review at conceptual level in order to identify the possibilities and potential problems involved. The meaning of different subareas of quality award criteria is analyzed from the point of view of R&D activities and single projects. The measures, performance criteria and concrete measurable aspects for R&D project evaluation are then derived on the basis of the analysis. In the empirical part of the study, the analysis of the applicability of derived measures for R&D project evaluation is discussed from the viewpoint of a manufacturing company that has successfully applied the Finnish National Quality Award Criteria. The main goal of the study is to promote effective R&D management by utilizing a systematic analysis approach based on the Quality Award Criteria framework in order to achieve better understanding of the meaning of each sub-area of the framework to the assessment of R&D as a whole and at the project level, as well as to propose new measurement subjects and evaluation methods or concrete measures for R&D projects. In order to evaluate the validity of the derivation results of the analysis approach and to fine-tune the results, the derived measurement subjects and potential R&D project measures are compared with the development needs of the R&D performance measurement in a case company. The systematic analysis approach provides new insights for companies to improve the performance of their R&D through the effective use of a quality award criteria framework. Generally, the results of the study promote the communication of criteria to the project level and taking different aspects
of the sub-areas of quality award criteria more strongly into account in systematic R&D projectlevel evaluation in different organizations. The study also gives examples of derivation of new R&D project metrics from the quality award criteria framework and comparison of the metrics in the light of the development needs of R&D performance measurement in a case company.
2. Quality management in R&D: A brief overview The quality of the R&D process is one of the critical success factors influencing the performance and success of a company’s R&D activities. According to a study by Cooper (1998), a high quality new product process is the strongest common denominator among high performance businesses. The effective management of R&D requires appropriate metrics for assessing the quality of the process. The principles of quality management have been applied in R&D management in several organizations, and their applicability has been reported in a number of earlier studies (e.g. Bellary and Murthy, 1999; Boyer, 1991; Fisher and Heywood, 1992; Kiella and Golhar, 1997; Lovett, 1992; Patino, 1997). Due to the nature of R&D activities—e.g. insecurity related to planning and decisionmaking, assessment of the contribution of R&D to profits, long time lag, creative personnel, coordination and control etc.—there are barriers hindering the effective use of these principles. 2.1. Quality management practices in R&D— theoretical considerations from previous studies Total Quality Management (TQM) can be defined as follows: ‘A business improvement philosophy which comprehensively and continuously involves all of an organization’s functions in improvement activities’ (Rosenau et al., 1996). TQM is an approach for improving the competitiveness, effectiveness and flexibility of a whole organization. It is essentially a way of planning, organizing and understanding each activity, and depends on each individual at each level (Oakland, 1993). The implications emerging from the total
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
quality movement embrace management and organizational theory and incorporate a wide range of quality strategies including customer satisfaction, benchmarking, time-based competition, process simplification, and performance evaluation. Quality management tools can be applied not only to production but also to costcutting, capital investment, environmental concerns, and research and development activities (Fisher and Heywood, 1992). The TQM philosophy has handed down useful legacies for R&D management, e.g. the understanding of customer needs, strengthening cross-functional and crossorganizational linkages and teamwork, formal benchmarking, measurement of R&D performance and the establishment of unifying business processes which integrated R&D into a broader enterprise-wide context (Chatterji and Davidson, 2001). Most TQM approaches strongly emphasize measurement, especially in the quality assurance and control areas (Oakland, 1993). According to Oakland (1993), the measurement system must be designed, planned and implemented to reflect customer requirements, give visibility to the processes and the progress made, communicate the total quality effort and engage a never ending improvement cycle. The introduction of the Malcolm Baldrige Quality Award in 1987 was a milestone in the evolution of total quality concepts in the US and
121
in the West in general. The launch of the European Quality Award (EQA) in 1991, which took the MBQA as its starting point, was a further advance in TQM development (Conti, 1997). This study focuses on the applicability of the Finnish National Quality Award Criteria, formerly based on the MBQA, but from the year 2001 based on the EFQM-model. Since the empirical part of the study describes the company that has successfully applied the criteria and participated in the Finnish National Quality Award contest in 1999, the framework for assessment in this study is based on the MBQA. A system perspective and causal chains between the sub-areas of the MBQA are presented in Fig. 1. The categories and weightings of MBQA criteria in the year 2000 are presented in Table 1. Self-assessment can be seen as a by-product of quality awards. The awards and self-assessments have different aims, and therefore the means of conducting assessments are different. For instance, Conti (1997) suggests to reverse the direction with respect to the traditional award assessment sequence, and to simply forget the weights in assessments, even though a ‘standard’ model like the MBQA is a wise choice for starting. The Finnish National Quality Award Criteria had the MBQA as the starting point until the year 2000. According to a survey made by the Center for Excellence—Finland (2001), the Finnish Quality Award is at least moderately well known
Fig. 1. Baldrige criteria for a performance excellence framework: a system perspective (National Institute of Standards and Technology, 2000).
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
122
Table 1 2000 Criteria for performance excellence—item listing (National Institute of Standards and Technology, 2000) 2000 categories/items 1
2
3
4
Point values
Leadership 1.1 Organizational leadership 1.2 Public responsibility and citizenship
85 40
Strategic planning 2.1 Strategy development 2.2 Strategy deployment
40 45
Customer and market focus 3.1 Customer and market knowledge 3.2 Customer satisfaction and relationships Information and analysis 4.1 Measurement of organizational performance 4.2 Analysis of organizational performance
125
85
85 40 45
85
45
Human resource focus 5.1 Work systems 35 5.2 Employee education, training, and 25 development 5.3 Employee well-being and satisfaction 25
85
6
Process management 6.1 Product and service processes 6.2 Support processes 6.3 Supplier and partnering processes
85 55 15 15
Business results 7.1 Customer focused results 7.2 Financial and market results 7.3 Human resource results 7.4 Supplier and partner results 7.5 Organizational effectiveness results
115 115 80 25 115
Total points
2.2. Quality management practices in R&D at Valtra Inc
40
5
7
One of the main differences between the Finnish National Quality Award Criteria and MBQA Criteria in the year 2000, was that there were eight categories in the Finnish Criteria. The eighth category included the issues of public responsibility and environmental effects. All the categories of the Quality Award Criteria also cover the areas of R&D, but they all cannot as such be utilized in R&D assessment. The development of R&D assessment with help of the Quality Award Criteria, however, makes it easier for R&D to be integrated in other activities of the company and its business development.
450
1000
among 43% of company managers in Finland. The quality award is best known in large companies, in which 72% of managers know the Finnish Quality Award well. 59% of these companies apply the quality award thinking and utilize it in their business development. The application of quality awards is increasing especially in small companies.
The empirical part of this study is based on the analysis of Valtra Inc, a business unit of the Partek Corporation. Valtra develops, manufactures, markets and services tractors for agricultural, forestry and municipal customers in more than 70 countries. It is the fifth biggest manufacturer of tractors in the world in its ranges. Valtra has put a lot of effort for long term quality management and continuous improvement, as well as taken many systematic steps to reach its quality goals. Valtra was granted the ISO 9001 quality certificate in 1993 as the first tractor manufacturer in the world, after comprehensive self-assessment and systematic development work. After this the quality development work was continued by applying the Malcolm Baldrige and Finnish Quality Award criteria as well as developing the ISO14001 environmental management system. The quality criteria and self-assessment have helped Valtra to clarify its biggest strengths and needed development areas as well as focus its quality development efforts. Valtra obtained the Finnish Quality Award in 1999 and certificated ISO14001-system in 2000. Valtra’s quality development work continues by applying QS9000 tools in its own operations and by training its suppliers to use the QS9000 tools and continuously develop their quality improvement capabilities. A clearly defined product development process is one of the key tools for quality management in
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
Valtra’s R&D. The product development process and other quality instructions are described in Valtra’s quality manual. The product development process, called product process, defines the needed phases, control points and tasks, and related documents for development projects. With the help of the product process the participants in the project and the steering group can check that all required tasks are fulfilled and risks are under control. One of the last tasks in each product process is the assessment of executed projects to clarify the success, strengths and weaknesses of their execution and to continuously improve the product process. The assessment is mainly based on discussions and collected experiences. Some quantitative criteria, like time and costs, are also examined, but new criteria to clarify the fulfillment of the goals of projects and the effectiveness of their execution are needed as well.
3. R&D performance assessment at the project level Corporate performance as well as R&D performance can be measured at several levels. Typical levels of assessment are company level, SBU level, department level, process level, project level and personal level. This study focuses on measurement aspects at the project level. However, assessment at the company or SBU level are also discussed, since many project level measures are derived from upper level measures. 3.1. Different measures for different R&D project stages R&D can be divided into stages in which different evaluation techniques are applied (see Pappas and Remer, 1985). Organizations often execute many different types of R&D projects, from fundamental basic research projects to product improvement projects. Therefore, both qualitative and quantitative evaluation techniques can be needed. Quantitative techniques usually follow a specific algorithm or predefined ratio to generate numbers that can be compared with other projects and past experiences. Semi-quantitative techniques are basically qualitative judgments that
123
are converted to numbers, and qualitative techniques are intuitive judgments (Pappas and Remer, 1985). According to Pappas and Remer (1985), qualitative techniques are best suited for basic research, semi-quantitative techniques for applied research and quantitative techniques for product development and improvements. To better understand how the metrics vary, Hauser and Zettelmeyer (1997) have introduced a tier metaphor, which enables us to categorize a diverse continuum of projects, programs and explorations and focus on key characteristics. ‘‘Tier 1’’ is defined as basic research that attempts to understand basic science and technology. Tier 1 explorations may have applicability to many business units or may spawn new business units. ‘‘Tier 2’’ is defined as those activities that select or develop programs to match the core technological competence of the organization. ‘‘Tier 3’’ is defined as specific projects focused on the more immediate needs of the customer, the business unit and/or the corporation. For applied projects, market outcome metrics are most relevant. In their study, Hauser and Zettelmeyer present R&D metrics, both qualitative judgments and quantitative measures, reported by interviewees, as well as their relevancy for the Tiers. Integrated metrics that contain an articulated but separable suite of quantitative and qualitative techniques can be flexibly applied across all types of R&D (Werner and Souder, 1997). Project-level measurement results can be utilized in several ways in R&D. Through measuring, activities can be better diagnosed, and significant problem areas or bottlenecks which influence the overall effectiveness of projects can be clearly detected. Cause and effect relationships in R&D project performance should be clarified in order to understand the significance of single projects for the whole R&D. The performance should be assessed in different phases of projects, and therefore both leading and lagging indicators are needed. Well-defined leading indicators can give valuable information and early signals predicting project success for project managers. The measures of project outputs reveal the quality and overall performance of the project. The measurement results can be utilized for instance in more effective
124
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
resource allocation and motivation of project personnel. Several different frameworks can promote the defining of appropriate R&D project measures. The MBQA framework is one of the possible framework, because it is well known and widely applied for company self-assessments. It is a systematic and flexible approach for defining the assessment criteria. 3.2. Development needs of R&D project measurements at Valtra Inc At the moment Valtra has five corporate level and three own measures for product development. The corporate level measures are: the average time to market of projects, changes in product costs, guarantee costs of products, the share of new products from total sales and the complexity of products (total number of items/total number of products). Valtra’s own R&D measures are the amount of new models per year, development of the average horsepower of produced tractors and the capability of R&D to keep defined project timetables at project level. The difference between the actual and planned project schedules affects the bonuses for the R&D personnel. The performance of projects is also evaluated by comparing the actual product costs and features with the cost target and project specifications, but there are no fixed systematically used measures for that. New measures are needed in Valtra to comprehensively clarify the fulfillment of project goals, product specifications and customer needs. New measures could be also useful in order to follow the quality of input information and the effectiveness of the execution of the different phases of projects. Creation of new innovative solutions and knowledge is becoming more and more important in Valtra’s R&D. Therefore, the R&D measures should also clarify innovativeness and development of knowledge both at project and departmental level. An essential development need in the measurement is to link the measures of different levels more accurately. Our approach to the derivation process of new potential R&D measures aims at enhancing this link. Valtra has also started to apply the main principles of the Balanced Scorecard approach
(Kaplan and Norton, 1996) to establish a new set of measures for the whole company as well as for R&D. Earlier it has utilized the Quality Award Criteria to support and focus company development. The Quality Award Criteria describe the success factors of the company’s processes as well as important results. Thus they also describe important measurement areas for R&D projects. Quality award criteria are utilized in the next chapter to derive new measures for manufacturing companies’ R&D projects.
4. Analysis of the utilization of quality award criteria in R&D project assessment In this chapter the application of quality award criteria for R&D project assessment is analyzed with the help of a systematic analysis approach. The analysis of the utilization of criteria for R&D project assessment is discussed from the viewpoint of a manufacturing company that has successfully applied the Finnish National Quality Award Criteria. In later implementation of the new measures derived on the basis of the analysis, also the most significant development needs related to the current R&D project metrics are carefully taken into account. Fig. 2 below presents a systematic analysis approach for the derivation of new R&D project measures with help of the MBQA framework. Each category involves several sub-areas (see Table 1), which are analyzed, but only the main categories are presented below. The meaning of each category is first described from the whole R&D perspective. Then the meaning of the category for projects and their assessments is analyzed. Several significant measurement subjects can then be derived from the results of the earlier phases of the analysis. In the last phase, new potential measures or evaluation methods are presented as a final result of the analysis utilizing the MBQA framework. The results of the analysis, i.e. the new measure proposals will then again be compared to the development needs related to the present R&D metrics and the whole measurement system. This comparison reveals the final applicability of the derived potential measures. The
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
MBQA Criteria
Meaning of criteria for the whole R&D
Meaning of criteria for R&D project assessment
Derived measurement subjects
Development needs of present R&D measures
125
Derived potential measures / evaluation methods
New measures for R&D project assessment
Fig. 2. The phased process of derivation of new R&D project-level measures.
derivation process was implemented in expert meetings with company representatives. The process was implemented systematically according to the phase-by-phase principle so that the derived potential measures were results from the fourth phase of the process. In Table 2 below the derivation process of R&D project-level measures in the Leadership-category is presented as an example. This example is a part of a much larger table. Similar derivations were implemented for all the seven categories, but their presentation in this context is not essential, as the principles and utilization of the derivation process are the focal point here. In the following table (Table 3), the examples of derivation results from different categories are presented. The example measures presented are chosen for their values of applicability and novelty from the case company’s point of view. The derived measures were compared with the development needs of the company in order to find the most applicable measures for R&D project assessment. In the following, we will evaluate the usability of the derived, Quality Award Criteria-based measures from the viewpoint of Valtra Inc. and its development needs in R&D performance measurement, which were presented in Section 3.2 above. New measures are needed for both function and project level measurement in order to support a continuous development of the effectiveness of R&D and its processes. At the project level new
measures are needed to clarify the fulfillment of project goals, specifications and customer needs as well as to follow the effectiveness of the execution and creation of new innovative solutions and knowledge. The executed derivation process produced several measurement areas and measures to meet the described development needs. Good new measures to clarify the fulfillment of customer needs, project goals and specifications are: *
*
*
*
*
*
Amount of realized customer requirements (‘‘must be’’ needs) and requests (‘‘should be’’ needs) stated in advanced executed customer surveys. Amount of features customers consider as their own (based on fulfillment of customer requirements). Performance in pilot customers’ satisfaction surveys. Amount of contacts between key customers, sales companies and personnel of the project. Share of forecasted and realized results of projects from the goals of the company and its R&D, for example share of stated increase in sales target (2/10%). Share of fully and partly fulfilled specifications.
The net present value of the project divided by its total working hours and development of goals, as well as the realized lead times of different phases of the product process in different projects could
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
126
Table 2 The derivation process of R&D project-level measures in the leadership-category MBQA criteria category
Meaning for company R&D
Meaning for R&D project assessment
Measurement subjects
How are the values known and ‘Value test’ of project internalized by project personnel, amount of ‘right’ personnel? answers
(1) Company values,
How are the factors taken into account in R&D projects. E.g. how is it confirmed that a project supports the (1) Company values,
(2) Innovative working environment
(2) Innovative working environment, and
1. Leadership How are the following factors taken into account in R&D by senior management?
Implementation of values?
Creativity, idea promotion?
Risk-taking capability? (3) Company’s direction (3) Direction of R&D and Contribution of a project to the whole company? the company objectives?
be useful measures to follow the effectiveness and efficiency of projects. The quality of input information could be followed by clarifying the reasons for changes of specifications and categorizing them into groups, and by following the share of different groups in the total amount of reasons. Innovativeness and learning are difficult areas to measure, but they are also becoming more and more important factors for the success of Valtra. Potential new measures defined in the derivation process for Valtra are:
*
*
*
*
Share of people who have produced new ideas from the total amount of personnel working in the project. Share of realized ideas out of the total amount of ideas presented in the project. Match (coverage %) between the needed knowledge/competence areas and available knowledge in the project. Assessment of critical competencies before and after the project.
*
Derived potential measures, evaluation methods
Post-project review of implementation of values, e.g. reliability: schedule-keeping Amount of initiatives Amount of people who made initiatives Amount of new ideas to be implemented Contribution of a project to the sales increase objectives Contribution of a project to gain new business segments
Number of days worked together with customers (for example in farms) in the project.
Measures that can be utilized at both the project and the functional level are practical to control the total amount of measures. Many of the above measures can be utilized at both levels. The executed derivation process has helped to create common measures for both the project and functional level as well as increased the understanding of links between functional level and project level measurement. The validity and reliability of the derived measures are assessed by the company management in the light of the development needs presented above.
5. Conclusions Approaches like the utilization of quality award criteria are widely recognized as good starting points for self-assessment in companies. Award assessments and self-assessments may, however,
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
127
Table 3 Examples of derivation results from different MBQA categories Leadership Amount of people who made initiatives Amount of ideas to be implemented per total amount of ideas Contribution of a project to the sales increase objectives Strategic planning Categorized amount of reasons for changes in a project Competence of project personnel vs. competence areas needed in a project (x% coverage) Project management assessment of the fulfillment of the strategy in a project (e.g. scale 1–5) Assessment of resources and strategic competencies before and after the project Customer and market focus Ability of project personnel to enumerate the main markets and main customers Contacts of project personnel and amount of visits to key customers’ sales offices Existence of collected customer need document (yes/no) and its clarity and scope Number of project people who know the specification and have the specification document Systematic satisfaction measurements of pilot customers Amount of recurred complaints (because of the same issue) Amount of realized customer requirements and requests stated in advanced executed customer surveys Number of features based on fulfillment of customer requirements Number of present customers who change to the product of project and the speed of change Information and analysis Availability of measurement information for project personnel Schedule objectives and schedule keeping in different phases of a project; Development of project lead time in similar types of projects (project efficiency index) Quality and amount index of communication Amount of problems/faults; prioritized most difficult problem areas Human resource focus Amount of initiatives per person Number of days in further education Development speed and level of a project team Process management Schedule keeping in a product process Amount of problems in planning and support systems per day per user Number of initiatives from subcontractors Business results New customers as a result of a project Project net present value per total working hours of a project Satisfaction of project partners Amount of new applied technologies Share of forecasted and realized results of projects from goals of company and its R&D Schedule keeping
have distinct purposes. When assessing and measuring performance at the R&D project level, the main purposes for assessing different types of projects need to be carefully clarified.
The aim of the research was to study the application of quality award criteria in the assessment of R&D projects by combining theory and practice so that new potential measures could
128
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
be derived with the help of our phased systematic analysis approach. The literature review helped to identify the possibilities and potential problems related to the issue at conceptual level. The meaning of different sub-areas of quality award criteria was analyzed from the point of view of R&D activities and single projects. New measures, performance criteria and concrete measurable aspects for R&D project evaluation were derived on the basis of the analysis. In practice, the analysis of the application of criteria for R&D project assessment was discussed from the viewpoint of a case company that has successfully applied the Quality Award Criteria in its business development. Example derivations of measures for R&D project assessments were described in the study. Similar types of new measures for project performance assessment were derived in the analysis of different categories of quality award criteria. For e.g. common measurements like schedule keeping, the satisfaction level of project personnel, customer satisfaction measurements, the achievement level of project objectives and number of initiatives of project personnel could be found in several categories. This reflects the basic principles—e.g. customer focus, concentration on results, innovation and learning—of quality management practices and quality awards. In addition to common assessment methods, which are often qualitative in nature, also more concrete, new potential measures were derived as a result of the systematic analysis approach. The executed derivation process helped to create common measures for both the project and functional level, and increased the understanding of links between functional level and project level measurement. Measures that can be utilized at both the project and functional level are practical to control the total amount of performance measures of a company’s R&D. Further studies within this issue concentrate on the utilization of the executed derivation approach and derived measures in companies’ R&D management and continuous improvement of R&D activities. The links between project and functional level measurements as well as the links between utilization of Quality Award Criteria framework and the Balanced Scorecard approach in compa-
nies’ self-assessment and performance measurement of R&D are also potential topics for deeper future studies.
References Bellary, A., Murthy, D.N.P., 1999. New product development process and total quality management. Proceedings of Portland International Conference on Management of Engineering and Technology. Portland, Oregon, USA. Boyer, S.M., 1991. Total quality management and new product development. Total Quality Management 2 (3), 283–289. The Center for Excellence—Finland (CEF), 2001. Suomen laatupalkinnon tunnettuus ja k.aytto. nousussa [Internet document], [referred 5.4.2002], Available at: http:// www.sly.fi/laatupalkinto/index64.html (in Finnish). Chatterji, D., Davidson, J.M., 2001. Examining TQM’s Legacies for R&D. Research Technology Management 44 (1), 10–12. Conti, T., 1997. Optimizing self-assessment. Total Quality Management 8 (2), 5–15. Cooper, R.G., 1998. Benchmarking new product performance: results of the best practices study. European Management Journal 16 (1), 1–17. Fisher, J., Heywood, C., 1992. Total quality management of Canadian R&D activities. CMA Magazine 66 (7), 25–28. Hauser, J., Zettelmeyer, F., 1997. Metrics to evaluate R, D&E. Research Technology Management 40 (4), 32–38. Kaplan, R.S., Norton, D.P., 1996. The Balanced Scorecard— Translating Strategy into Action. Harvard Business School Press, Boston. Kiella, M.L., Golhar, D.Y., 1997. Total quality management in an R&D environment. International Journal of Operations and Production Management 17 (2), 184–198. Lovett, J.R., 1992. Doing the right things right. All the time. Research Technology Management 35 (5), 35–38. National Institute of Standards and Technology, 2000. Criteria for Performance Excellence [pdf document], [referred 5.4.2002], Available at: http://www.quality.nist.gov/ PDF files/2000 Business Criteria.pdf. Oakland, J.S., 1993. Total Quality Management—the Route to Improving Performance, 2nd Edition. Butterworth-Heinemann, Oxford. Pappas, R.A., Remer, D.S., 1985. Measuring R&D productivity. Research Management 28 (3), 15–22. Patino, H., 1997. Applying total quality to R&D at Coors Brewing Company. Research Technology Management 40 (5), 32–36. Rosenau, M.D., Jr., A. Griffin, Castellion, G.A., Anschuetz, N.F. (Eds.), 1996. The PDMA Handbook of New Product Development. John Wiley & Sons, New York. Werner, B., Souder, W., 1997. Measuring R&D performance— state of the art. Research Technology Management 40 (2), 34–42.
Applying quality award criteria in R&D project assessment Ville Ojanena,*, Petteri Piippob, Markku Tuominena a
Department of Industrial Engineering and Management, Lappeenranta University of Technology, P.O. Box 20, FIN-53851 Lappeenranta, Finland b R&D Center, Valtra Inc., P.O. Box 557, 40101 Jyvaskyl a, . . Finland
Abstract Due to the problematic characteristics and complexity of R&D, the applicability of Total Quality Management (TQM) and quality award criteria in the assessment of a company’s R&D process is very challenging. This study concentrates especially on the application of the Malcolm Baldrige Quality Award (MBQA) Criteria to R&D assessments at project level. The applicability of quality award criteria in the assessment of R&D projects is first discussed with the help of a literature review at conceptual level. The meaning of different sub-areas of quality award criteria is analyzed from the point of view of R&D activities and single projects. The measures, performance criteria and concrete measurable aspects for R&D project evaluation are then derived on the basis of the analysis. In the empirical part of the study, the analysis of the utilization of criteria for R&D project assessment is discussed from the viewpoint of a manufacturing company that has successfully applied the Finnish National Quality Award Criteria based on the MBQA. The study gives examples of the derivation of new R&D project measures from the quality award criteria framework. r 2002 Elsevier Science B.V. All rights reserved. Keywords: Total quality management (TQM); Quality award criteria; Self-assessment; R&D projects; Performance evaluation
1. Introduction An important consequence of the introduction of systematic approaches of quality management, e.g. quality awards, is the increasingly widespread use of their models and criteria for company selfassessment. Several similar types of national criteria are used for assessment, e.g. the Malcolm Baldrige Quality Award Criteria, the European Foundation for Quality Management (EFQM)model, the Deming Prize and the Finnish National *Corresponding author. Tel.: +358-5621-2670; fax: +3585621-2667. E-mail address: ville.ojanen@lut.fi (V. Ojanen).
Quality Award Criteria. This study concentrates on application of Finnish criteria to R&D project assessments. The Finnish Criteria were formerly based on the MBQA, but from the year 2001 they are based on the EFQM-model. The framework utilized in this study is based on the MBQA. In a number of earlier studies, the Total Quality Management (TQM) philosophy has been argued to be an applicable approach also for the management of R&D. However, due to the problematic characteristics and complexity of R&D, the application of quality award criteria in the assessment of a company’s R&D process is very challenging (see e.g. Bellary and Murthy, 1999; Boyer, 1991; Fisher and Heywood, 1992; Kiella
0925-5273/02/$ - see front matter r 2002 Elsevier Science B.V. All rights reserved. PII: S 0 9 2 5 - 5 2 7 3 ( 0 2 ) 0 0 2 4 7 - 5
120
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
and Golhar, 1997; Lovett, 1992; Patino, 1997). Companies have different types of R&D projects and in addition to the process level, suitable performance criteria are needed also at the project level. There is a need in many organizations to bring the criteria also to project-level assessment in order to effectively control and manage different types of single R&D projects and the whole project portfolio. The proposed systematic analysis approach of this study has several phases. First, the application of quality award criteria in the assessment and evaluation of R&D projects is discussed with the help of a literature review at conceptual level in order to identify the possibilities and potential problems involved. The meaning of different subareas of quality award criteria is analyzed from the point of view of R&D activities and single projects. The measures, performance criteria and concrete measurable aspects for R&D project evaluation are then derived on the basis of the analysis. In the empirical part of the study, the analysis of the applicability of derived measures for R&D project evaluation is discussed from the viewpoint of a manufacturing company that has successfully applied the Finnish National Quality Award Criteria. The main goal of the study is to promote effective R&D management by utilizing a systematic analysis approach based on the Quality Award Criteria framework in order to achieve better understanding of the meaning of each sub-area of the framework to the assessment of R&D as a whole and at the project level, as well as to propose new measurement subjects and evaluation methods or concrete measures for R&D projects. In order to evaluate the validity of the derivation results of the analysis approach and to fine-tune the results, the derived measurement subjects and potential R&D project measures are compared with the development needs of the R&D performance measurement in a case company. The systematic analysis approach provides new insights for companies to improve the performance of their R&D through the effective use of a quality award criteria framework. Generally, the results of the study promote the communication of criteria to the project level and taking different aspects
of the sub-areas of quality award criteria more strongly into account in systematic R&D projectlevel evaluation in different organizations. The study also gives examples of derivation of new R&D project metrics from the quality award criteria framework and comparison of the metrics in the light of the development needs of R&D performance measurement in a case company.
2. Quality management in R&D: A brief overview The quality of the R&D process is one of the critical success factors influencing the performance and success of a company’s R&D activities. According to a study by Cooper (1998), a high quality new product process is the strongest common denominator among high performance businesses. The effective management of R&D requires appropriate metrics for assessing the quality of the process. The principles of quality management have been applied in R&D management in several organizations, and their applicability has been reported in a number of earlier studies (e.g. Bellary and Murthy, 1999; Boyer, 1991; Fisher and Heywood, 1992; Kiella and Golhar, 1997; Lovett, 1992; Patino, 1997). Due to the nature of R&D activities—e.g. insecurity related to planning and decisionmaking, assessment of the contribution of R&D to profits, long time lag, creative personnel, coordination and control etc.—there are barriers hindering the effective use of these principles. 2.1. Quality management practices in R&D— theoretical considerations from previous studies Total Quality Management (TQM) can be defined as follows: ‘A business improvement philosophy which comprehensively and continuously involves all of an organization’s functions in improvement activities’ (Rosenau et al., 1996). TQM is an approach for improving the competitiveness, effectiveness and flexibility of a whole organization. It is essentially a way of planning, organizing and understanding each activity, and depends on each individual at each level (Oakland, 1993). The implications emerging from the total
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
quality movement embrace management and organizational theory and incorporate a wide range of quality strategies including customer satisfaction, benchmarking, time-based competition, process simplification, and performance evaluation. Quality management tools can be applied not only to production but also to costcutting, capital investment, environmental concerns, and research and development activities (Fisher and Heywood, 1992). The TQM philosophy has handed down useful legacies for R&D management, e.g. the understanding of customer needs, strengthening cross-functional and crossorganizational linkages and teamwork, formal benchmarking, measurement of R&D performance and the establishment of unifying business processes which integrated R&D into a broader enterprise-wide context (Chatterji and Davidson, 2001). Most TQM approaches strongly emphasize measurement, especially in the quality assurance and control areas (Oakland, 1993). According to Oakland (1993), the measurement system must be designed, planned and implemented to reflect customer requirements, give visibility to the processes and the progress made, communicate the total quality effort and engage a never ending improvement cycle. The introduction of the Malcolm Baldrige Quality Award in 1987 was a milestone in the evolution of total quality concepts in the US and
121
in the West in general. The launch of the European Quality Award (EQA) in 1991, which took the MBQA as its starting point, was a further advance in TQM development (Conti, 1997). This study focuses on the applicability of the Finnish National Quality Award Criteria, formerly based on the MBQA, but from the year 2001 based on the EFQM-model. Since the empirical part of the study describes the company that has successfully applied the criteria and participated in the Finnish National Quality Award contest in 1999, the framework for assessment in this study is based on the MBQA. A system perspective and causal chains between the sub-areas of the MBQA are presented in Fig. 1. The categories and weightings of MBQA criteria in the year 2000 are presented in Table 1. Self-assessment can be seen as a by-product of quality awards. The awards and self-assessments have different aims, and therefore the means of conducting assessments are different. For instance, Conti (1997) suggests to reverse the direction with respect to the traditional award assessment sequence, and to simply forget the weights in assessments, even though a ‘standard’ model like the MBQA is a wise choice for starting. The Finnish National Quality Award Criteria had the MBQA as the starting point until the year 2000. According to a survey made by the Center for Excellence—Finland (2001), the Finnish Quality Award is at least moderately well known
Fig. 1. Baldrige criteria for a performance excellence framework: a system perspective (National Institute of Standards and Technology, 2000).
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
122
Table 1 2000 Criteria for performance excellence—item listing (National Institute of Standards and Technology, 2000) 2000 categories/items 1
2
3
4
Point values
Leadership 1.1 Organizational leadership 1.2 Public responsibility and citizenship
85 40
Strategic planning 2.1 Strategy development 2.2 Strategy deployment
40 45
Customer and market focus 3.1 Customer and market knowledge 3.2 Customer satisfaction and relationships Information and analysis 4.1 Measurement of organizational performance 4.2 Analysis of organizational performance
125
85
85 40 45
85
45
Human resource focus 5.1 Work systems 35 5.2 Employee education, training, and 25 development 5.3 Employee well-being and satisfaction 25
85
6
Process management 6.1 Product and service processes 6.2 Support processes 6.3 Supplier and partnering processes
85 55 15 15
Business results 7.1 Customer focused results 7.2 Financial and market results 7.3 Human resource results 7.4 Supplier and partner results 7.5 Organizational effectiveness results
115 115 80 25 115
Total points
2.2. Quality management practices in R&D at Valtra Inc
40
5
7
One of the main differences between the Finnish National Quality Award Criteria and MBQA Criteria in the year 2000, was that there were eight categories in the Finnish Criteria. The eighth category included the issues of public responsibility and environmental effects. All the categories of the Quality Award Criteria also cover the areas of R&D, but they all cannot as such be utilized in R&D assessment. The development of R&D assessment with help of the Quality Award Criteria, however, makes it easier for R&D to be integrated in other activities of the company and its business development.
450
1000
among 43% of company managers in Finland. The quality award is best known in large companies, in which 72% of managers know the Finnish Quality Award well. 59% of these companies apply the quality award thinking and utilize it in their business development. The application of quality awards is increasing especially in small companies.
The empirical part of this study is based on the analysis of Valtra Inc, a business unit of the Partek Corporation. Valtra develops, manufactures, markets and services tractors for agricultural, forestry and municipal customers in more than 70 countries. It is the fifth biggest manufacturer of tractors in the world in its ranges. Valtra has put a lot of effort for long term quality management and continuous improvement, as well as taken many systematic steps to reach its quality goals. Valtra was granted the ISO 9001 quality certificate in 1993 as the first tractor manufacturer in the world, after comprehensive self-assessment and systematic development work. After this the quality development work was continued by applying the Malcolm Baldrige and Finnish Quality Award criteria as well as developing the ISO14001 environmental management system. The quality criteria and self-assessment have helped Valtra to clarify its biggest strengths and needed development areas as well as focus its quality development efforts. Valtra obtained the Finnish Quality Award in 1999 and certificated ISO14001-system in 2000. Valtra’s quality development work continues by applying QS9000 tools in its own operations and by training its suppliers to use the QS9000 tools and continuously develop their quality improvement capabilities. A clearly defined product development process is one of the key tools for quality management in
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
Valtra’s R&D. The product development process and other quality instructions are described in Valtra’s quality manual. The product development process, called product process, defines the needed phases, control points and tasks, and related documents for development projects. With the help of the product process the participants in the project and the steering group can check that all required tasks are fulfilled and risks are under control. One of the last tasks in each product process is the assessment of executed projects to clarify the success, strengths and weaknesses of their execution and to continuously improve the product process. The assessment is mainly based on discussions and collected experiences. Some quantitative criteria, like time and costs, are also examined, but new criteria to clarify the fulfillment of the goals of projects and the effectiveness of their execution are needed as well.
3. R&D performance assessment at the project level Corporate performance as well as R&D performance can be measured at several levels. Typical levels of assessment are company level, SBU level, department level, process level, project level and personal level. This study focuses on measurement aspects at the project level. However, assessment at the company or SBU level are also discussed, since many project level measures are derived from upper level measures. 3.1. Different measures for different R&D project stages R&D can be divided into stages in which different evaluation techniques are applied (see Pappas and Remer, 1985). Organizations often execute many different types of R&D projects, from fundamental basic research projects to product improvement projects. Therefore, both qualitative and quantitative evaluation techniques can be needed. Quantitative techniques usually follow a specific algorithm or predefined ratio to generate numbers that can be compared with other projects and past experiences. Semi-quantitative techniques are basically qualitative judgments that
123
are converted to numbers, and qualitative techniques are intuitive judgments (Pappas and Remer, 1985). According to Pappas and Remer (1985), qualitative techniques are best suited for basic research, semi-quantitative techniques for applied research and quantitative techniques for product development and improvements. To better understand how the metrics vary, Hauser and Zettelmeyer (1997) have introduced a tier metaphor, which enables us to categorize a diverse continuum of projects, programs and explorations and focus on key characteristics. ‘‘Tier 1’’ is defined as basic research that attempts to understand basic science and technology. Tier 1 explorations may have applicability to many business units or may spawn new business units. ‘‘Tier 2’’ is defined as those activities that select or develop programs to match the core technological competence of the organization. ‘‘Tier 3’’ is defined as specific projects focused on the more immediate needs of the customer, the business unit and/or the corporation. For applied projects, market outcome metrics are most relevant. In their study, Hauser and Zettelmeyer present R&D metrics, both qualitative judgments and quantitative measures, reported by interviewees, as well as their relevancy for the Tiers. Integrated metrics that contain an articulated but separable suite of quantitative and qualitative techniques can be flexibly applied across all types of R&D (Werner and Souder, 1997). Project-level measurement results can be utilized in several ways in R&D. Through measuring, activities can be better diagnosed, and significant problem areas or bottlenecks which influence the overall effectiveness of projects can be clearly detected. Cause and effect relationships in R&D project performance should be clarified in order to understand the significance of single projects for the whole R&D. The performance should be assessed in different phases of projects, and therefore both leading and lagging indicators are needed. Well-defined leading indicators can give valuable information and early signals predicting project success for project managers. The measures of project outputs reveal the quality and overall performance of the project. The measurement results can be utilized for instance in more effective
124
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
resource allocation and motivation of project personnel. Several different frameworks can promote the defining of appropriate R&D project measures. The MBQA framework is one of the possible framework, because it is well known and widely applied for company self-assessments. It is a systematic and flexible approach for defining the assessment criteria. 3.2. Development needs of R&D project measurements at Valtra Inc At the moment Valtra has five corporate level and three own measures for product development. The corporate level measures are: the average time to market of projects, changes in product costs, guarantee costs of products, the share of new products from total sales and the complexity of products (total number of items/total number of products). Valtra’s own R&D measures are the amount of new models per year, development of the average horsepower of produced tractors and the capability of R&D to keep defined project timetables at project level. The difference between the actual and planned project schedules affects the bonuses for the R&D personnel. The performance of projects is also evaluated by comparing the actual product costs and features with the cost target and project specifications, but there are no fixed systematically used measures for that. New measures are needed in Valtra to comprehensively clarify the fulfillment of project goals, product specifications and customer needs. New measures could be also useful in order to follow the quality of input information and the effectiveness of the execution of the different phases of projects. Creation of new innovative solutions and knowledge is becoming more and more important in Valtra’s R&D. Therefore, the R&D measures should also clarify innovativeness and development of knowledge both at project and departmental level. An essential development need in the measurement is to link the measures of different levels more accurately. Our approach to the derivation process of new potential R&D measures aims at enhancing this link. Valtra has also started to apply the main principles of the Balanced Scorecard approach
(Kaplan and Norton, 1996) to establish a new set of measures for the whole company as well as for R&D. Earlier it has utilized the Quality Award Criteria to support and focus company development. The Quality Award Criteria describe the success factors of the company’s processes as well as important results. Thus they also describe important measurement areas for R&D projects. Quality award criteria are utilized in the next chapter to derive new measures for manufacturing companies’ R&D projects.
4. Analysis of the utilization of quality award criteria in R&D project assessment In this chapter the application of quality award criteria for R&D project assessment is analyzed with the help of a systematic analysis approach. The analysis of the utilization of criteria for R&D project assessment is discussed from the viewpoint of a manufacturing company that has successfully applied the Finnish National Quality Award Criteria. In later implementation of the new measures derived on the basis of the analysis, also the most significant development needs related to the current R&D project metrics are carefully taken into account. Fig. 2 below presents a systematic analysis approach for the derivation of new R&D project measures with help of the MBQA framework. Each category involves several sub-areas (see Table 1), which are analyzed, but only the main categories are presented below. The meaning of each category is first described from the whole R&D perspective. Then the meaning of the category for projects and their assessments is analyzed. Several significant measurement subjects can then be derived from the results of the earlier phases of the analysis. In the last phase, new potential measures or evaluation methods are presented as a final result of the analysis utilizing the MBQA framework. The results of the analysis, i.e. the new measure proposals will then again be compared to the development needs related to the present R&D metrics and the whole measurement system. This comparison reveals the final applicability of the derived potential measures. The
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
MBQA Criteria
Meaning of criteria for the whole R&D
Meaning of criteria for R&D project assessment
Derived measurement subjects
Development needs of present R&D measures
125
Derived potential measures / evaluation methods
New measures for R&D project assessment
Fig. 2. The phased process of derivation of new R&D project-level measures.
derivation process was implemented in expert meetings with company representatives. The process was implemented systematically according to the phase-by-phase principle so that the derived potential measures were results from the fourth phase of the process. In Table 2 below the derivation process of R&D project-level measures in the Leadership-category is presented as an example. This example is a part of a much larger table. Similar derivations were implemented for all the seven categories, but their presentation in this context is not essential, as the principles and utilization of the derivation process are the focal point here. In the following table (Table 3), the examples of derivation results from different categories are presented. The example measures presented are chosen for their values of applicability and novelty from the case company’s point of view. The derived measures were compared with the development needs of the company in order to find the most applicable measures for R&D project assessment. In the following, we will evaluate the usability of the derived, Quality Award Criteria-based measures from the viewpoint of Valtra Inc. and its development needs in R&D performance measurement, which were presented in Section 3.2 above. New measures are needed for both function and project level measurement in order to support a continuous development of the effectiveness of R&D and its processes. At the project level new
measures are needed to clarify the fulfillment of project goals, specifications and customer needs as well as to follow the effectiveness of the execution and creation of new innovative solutions and knowledge. The executed derivation process produced several measurement areas and measures to meet the described development needs. Good new measures to clarify the fulfillment of customer needs, project goals and specifications are: *
*
*
*
*
*
Amount of realized customer requirements (‘‘must be’’ needs) and requests (‘‘should be’’ needs) stated in advanced executed customer surveys. Amount of features customers consider as their own (based on fulfillment of customer requirements). Performance in pilot customers’ satisfaction surveys. Amount of contacts between key customers, sales companies and personnel of the project. Share of forecasted and realized results of projects from the goals of the company and its R&D, for example share of stated increase in sales target (2/10%). Share of fully and partly fulfilled specifications.
The net present value of the project divided by its total working hours and development of goals, as well as the realized lead times of different phases of the product process in different projects could
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
126
Table 2 The derivation process of R&D project-level measures in the leadership-category MBQA criteria category
Meaning for company R&D
Meaning for R&D project assessment
Measurement subjects
How are the values known and ‘Value test’ of project internalized by project personnel, amount of ‘right’ personnel? answers
(1) Company values,
How are the factors taken into account in R&D projects. E.g. how is it confirmed that a project supports the (1) Company values,
(2) Innovative working environment
(2) Innovative working environment, and
1. Leadership How are the following factors taken into account in R&D by senior management?
Implementation of values?
Creativity, idea promotion?
Risk-taking capability? (3) Company’s direction (3) Direction of R&D and Contribution of a project to the whole company? the company objectives?
be useful measures to follow the effectiveness and efficiency of projects. The quality of input information could be followed by clarifying the reasons for changes of specifications and categorizing them into groups, and by following the share of different groups in the total amount of reasons. Innovativeness and learning are difficult areas to measure, but they are also becoming more and more important factors for the success of Valtra. Potential new measures defined in the derivation process for Valtra are:
*
*
*
*
Share of people who have produced new ideas from the total amount of personnel working in the project. Share of realized ideas out of the total amount of ideas presented in the project. Match (coverage %) between the needed knowledge/competence areas and available knowledge in the project. Assessment of critical competencies before and after the project.
*
Derived potential measures, evaluation methods
Post-project review of implementation of values, e.g. reliability: schedule-keeping Amount of initiatives Amount of people who made initiatives Amount of new ideas to be implemented Contribution of a project to the sales increase objectives Contribution of a project to gain new business segments
Number of days worked together with customers (for example in farms) in the project.
Measures that can be utilized at both the project and the functional level are practical to control the total amount of measures. Many of the above measures can be utilized at both levels. The executed derivation process has helped to create common measures for both the project and functional level as well as increased the understanding of links between functional level and project level measurement. The validity and reliability of the derived measures are assessed by the company management in the light of the development needs presented above.
5. Conclusions Approaches like the utilization of quality award criteria are widely recognized as good starting points for self-assessment in companies. Award assessments and self-assessments may, however,
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
127
Table 3 Examples of derivation results from different MBQA categories Leadership Amount of people who made initiatives Amount of ideas to be implemented per total amount of ideas Contribution of a project to the sales increase objectives Strategic planning Categorized amount of reasons for changes in a project Competence of project personnel vs. competence areas needed in a project (x% coverage) Project management assessment of the fulfillment of the strategy in a project (e.g. scale 1–5) Assessment of resources and strategic competencies before and after the project Customer and market focus Ability of project personnel to enumerate the main markets and main customers Contacts of project personnel and amount of visits to key customers’ sales offices Existence of collected customer need document (yes/no) and its clarity and scope Number of project people who know the specification and have the specification document Systematic satisfaction measurements of pilot customers Amount of recurred complaints (because of the same issue) Amount of realized customer requirements and requests stated in advanced executed customer surveys Number of features based on fulfillment of customer requirements Number of present customers who change to the product of project and the speed of change Information and analysis Availability of measurement information for project personnel Schedule objectives and schedule keeping in different phases of a project; Development of project lead time in similar types of projects (project efficiency index) Quality and amount index of communication Amount of problems/faults; prioritized most difficult problem areas Human resource focus Amount of initiatives per person Number of days in further education Development speed and level of a project team Process management Schedule keeping in a product process Amount of problems in planning and support systems per day per user Number of initiatives from subcontractors Business results New customers as a result of a project Project net present value per total working hours of a project Satisfaction of project partners Amount of new applied technologies Share of forecasted and realized results of projects from goals of company and its R&D Schedule keeping
have distinct purposes. When assessing and measuring performance at the R&D project level, the main purposes for assessing different types of projects need to be carefully clarified.
The aim of the research was to study the application of quality award criteria in the assessment of R&D projects by combining theory and practice so that new potential measures could
128
V. Ojanen et al. / Int. J. Production Economics 80 (2002) 119–128
be derived with the help of our phased systematic analysis approach. The literature review helped to identify the possibilities and potential problems related to the issue at conceptual level. The meaning of different sub-areas of quality award criteria was analyzed from the point of view of R&D activities and single projects. New measures, performance criteria and concrete measurable aspects for R&D project evaluation were derived on the basis of the analysis. In practice, the analysis of the application of criteria for R&D project assessment was discussed from the viewpoint of a case company that has successfully applied the Quality Award Criteria in its business development. Example derivations of measures for R&D project assessments were described in the study. Similar types of new measures for project performance assessment were derived in the analysis of different categories of quality award criteria. For e.g. common measurements like schedule keeping, the satisfaction level of project personnel, customer satisfaction measurements, the achievement level of project objectives and number of initiatives of project personnel could be found in several categories. This reflects the basic principles—e.g. customer focus, concentration on results, innovation and learning—of quality management practices and quality awards. In addition to common assessment methods, which are often qualitative in nature, also more concrete, new potential measures were derived as a result of the systematic analysis approach. The executed derivation process helped to create common measures for both the project and functional level, and increased the understanding of links between functional level and project level measurement. Measures that can be utilized at both the project and functional level are practical to control the total amount of performance measures of a company’s R&D. Further studies within this issue concentrate on the utilization of the executed derivation approach and derived measures in companies’ R&D management and continuous improvement of R&D activities. The links between project and functional level measurements as well as the links between utilization of Quality Award Criteria framework and the Balanced Scorecard approach in compa-
nies’ self-assessment and performance measurement of R&D are also potential topics for deeper future studies.
References Bellary, A., Murthy, D.N.P., 1999. New product development process and total quality management. Proceedings of Portland International Conference on Management of Engineering and Technology. Portland, Oregon, USA. Boyer, S.M., 1991. Total quality management and new product development. Total Quality Management 2 (3), 283–289. The Center for Excellence—Finland (CEF), 2001. Suomen laatupalkinnon tunnettuus ja k.aytto. nousussa [Internet document], [referred 5.4.2002], Available at: http:// www.sly.fi/laatupalkinto/index64.html (in Finnish). Chatterji, D., Davidson, J.M., 2001. Examining TQM’s Legacies for R&D. Research Technology Management 44 (1), 10–12. Conti, T., 1997. Optimizing self-assessment. Total Quality Management 8 (2), 5–15. Cooper, R.G., 1998. Benchmarking new product performance: results of the best practices study. European Management Journal 16 (1), 1–17. Fisher, J., Heywood, C., 1992. Total quality management of Canadian R&D activities. CMA Magazine 66 (7), 25–28. Hauser, J., Zettelmeyer, F., 1997. Metrics to evaluate R, D&E. Research Technology Management 40 (4), 32–38. Kaplan, R.S., Norton, D.P., 1996. The Balanced Scorecard— Translating Strategy into Action. Harvard Business School Press, Boston. Kiella, M.L., Golhar, D.Y., 1997. Total quality management in an R&D environment. International Journal of Operations and Production Management 17 (2), 184–198. Lovett, J.R., 1992. Doing the right things right. All the time. Research Technology Management 35 (5), 35–38. National Institute of Standards and Technology, 2000. Criteria for Performance Excellence [pdf document], [referred 5.4.2002], Available at: http://www.quality.nist.gov/ PDF files/2000 Business Criteria.pdf. Oakland, J.S., 1993. Total Quality Management—the Route to Improving Performance, 2nd Edition. Butterworth-Heinemann, Oxford. Pappas, R.A., Remer, D.S., 1985. Measuring R&D productivity. Research Management 28 (3), 15–22. Patino, H., 1997. Applying total quality to R&D at Coors Brewing Company. Research Technology Management 40 (5), 32–36. Rosenau, M.D., Jr., A. Griffin, Castellion, G.A., Anschuetz, N.F. (Eds.), 1996. The PDMA Handbook of New Product Development. John Wiley & Sons, New York. Werner, B., Souder, W., 1997. Measuring R&D performance— state of the art. Research Technology Management 40 (2), 34–42.